@heripo/research-radar 5.0.5 → 5.1.0

package/README.md

@@ -15,15 +15,15 @@ English | [한국어](./README-ko.md)
 
 An AI-powered newsletter service for Korean cultural heritage. Built on [`@llm-newsletter-kit/core`](https://github.com/heripo-lab/llm-newsletter-kit-core), it's both a production service ([live at heripo.app](https://heripo.app/research-radar/subscribe)) and a reference implementation showing how to build automated newsletters with LLMs.
 
-**Production metrics**:
-- **Cost**: $0.2-1 USD per issue
-- **Operation**: Fully autonomous 24/7 (no human intervention)
-- **Engagement**: 15% CTR
+This package provides crawling, analysis, content generation, and email templates. The hosted service supplies database repositories, scheduling, subscriber management, and email delivery separately.
+
+Previously reported service metrics were $0.2–1 per issue and 15% CTR. These are historical figures, not benchmarks for the current models or deployment.
 
 **Technical highlights**:
+
 - Type-safe TypeScript with strict interfaces
 - Provider pattern for swapping components (Crawling/Analysis/Content/Email)
-- 66 crawling targets across heritage agencies, museums, academic societies
+- 73 active crawling targets across heritage agencies, museums, academic societies, filtered at runtime by robots.txt
 - Multi LLM providers: OpenAI GPT-5 (analysis) + selectable content generation (OpenAI / Anthropic / Google)
 - Built-in retries, chain options, preview emails
 
@@ -33,7 +33,7 @@ An AI-powered newsletter service for Korean cultural heritage. Built on [`@llm-n
 
 Created by archaeologist-turned-engineer Hongyeon Kim to answer: "Why must research rely on labor-intensive manual work?"
 
-A personal script evolved into a production service after completing research on [Archaeological Informatization Using LLMs](https://poc.heripo.org). This repository open-sources the running service so developers can build 
+A personal script evolved into a production service after completing research on [Archaeological Informatization Using LLMs](https://poc.heripo.org). This repository open-sources the running service so developers can build
 domain-specific newsletters without starting from scratch.
 
 ## License
@@ -67,63 +67,88 @@ For academic publications:
 ## Installation
 
 ```bash
-npm install @heripo/research-radar @llm-newsletter-kit/core
+npm install @heripo/research-radar '@llm-newsletter-kit/core@~3.0.0'
 ```
 
-**Requirements**: Node.js >= 24, OpenAI API key, content generation API key (OpenAI / Anthropic / Google)
+**Requirements**: Node.js >= 24 and an ESM application. The package exports `dist/index.js` and TypeScript declarations (`dist/index.d.ts`), with a JavaScript sourcemap. The core engine is a peer dependency; the current supported range is `~3.0.5`. 3.0.5 is the floor because the newsletter generation prompt this package ships relies on the self-verification retry cap added there.
 
-**Note**: `@llm-newsletter-kit/core` is a peer dependency and must be installed separately.
+Article analysis requires an OpenAI API key. Content generation requires a key for the selected provider (OpenAI / Anthropic / Google); OpenAI can use the same key for both. Keys are passed explicitly to the library; load environment variables in your application.
 
 ## Quick Start
 
+Implement the four repository interfaces for your storage layer, then call this application-level wrapper. The repositories in this example are supplied by the caller; no database adapter is bundled.
+
 ```typescript
-import { generateNewsletter } from '@heripo/research-radar';
-
-const newsletterId = await generateNewsletter({
-  openAIApiKey: process.env.OPENAI_API_KEY,
-  contentGeneration: {
-    provider: 'anthropic',           // 'openai' | 'anthropic' | 'google'
-    apiKey: process.env.ANTHROPIC_API_KEY,
-    // model: 'claude-sonnet-4-6',   // optional, uses sensible default
-  },
+import {
+  type ArticleRepository,
+  type NewsletterRepository,
+  type TagRepository,
+  type TaskRepository,
+  generateNewsletter,
+} from '@heripo/research-radar';
+
+export async function runNewsletter(repositories: {
+  taskRepository: TaskRepository;
+  articleRepository: ArticleRepository;
+  tagRepository: TagRepository;
+  newsletterRepository: NewsletterRepository;
+}) {
+  const apiKey = process.env.OPENAI_API_KEY;
+  if (!apiKey) throw new Error('OPENAI_API_KEY is required');
+
+  const newsletterId = await generateNewsletter({
+    ...repositories,
+    openAIApiKey: apiKey,
+    contentGeneration: { provider: 'openai', apiKey },
+    logger: console,
+  });
+
+  if (newsletterId === null) {
+    console.log('No newsletter was created for this run.');
+    return null;
+  }
+
+  console.log('Saved newsletter:', newsletterId);
+  return newsletterId;
+}
+```
 
-  // Implement these repository interfaces (see src/types/dependencies.ts)
-  taskRepository: {
-    createTask: async () => db.tasks.create({ status: 'running' }),
-    completeTask: async (id) => db.tasks.update(id, { status: 'completed' }),
-  },
+`generateNewsletter()` returns `Promise<string | number | null>`: the saved newsletter ID, or `null` when no issue is created (for example, when publication criteria are not met). Generation errors can reject the promise. The internal `createNewsletterGenerator()` factory is not exported.
 
-  articleRepository: {
-    findByUrls: async (urls) => db.articles.findByUrls(urls),
-    saveCrawledArticles: async (articles, ctx) => db.articles.save(articles, ctx),
-    findUnscoredArticles: async () => db.articles.findUnscored(),
-    updateAnalysis: async (article) => db.articles.updateAnalysis(article),
-    findCandidatesForNewsletter: async () => db.articles.findCandidates(),
-  },
+### Repository contracts
 
-  tagRepository: {
-    findAllTags: async () => db.tags.findAll(),
-  },
+All interfaces are exported from the package root and defined in [src/types/dependencies.ts](./src/types/dependencies.ts). Article and newsletter data types come from `@llm-newsletter-kit/core`.
 
-  newsletterRepository: {
-    getNextIssueOrder: async () => db.newsletters.getNextOrder(),
-    saveNewsletter: async (data) => db.newsletters.save(data),
-  },
+| Repository             | Method                                         | Required result                                                                  |
+| ---------------------- | ---------------------------------------------- | -------------------------------------------------------------------------------- |
+| `TaskRepository`       | `createTask()`                                 | `Promise<number>` — task ID, not a task record                                   |
+|                        | `completeTask(taskId)`                         | `Promise<void>`                                                                  |
+| `ArticleRepository`    | `findByUrls(urls)`                             | `Promise<ParsedTarget[]>` for deduplication                                      |
+|                        | `saveCrawledArticles(articles, context)`       | `Promise<number>` — saved count; preserve task, target group, and target context |
+|                        | `findUnscoredArticles()`                       | `Promise<UnscoredArticle[]>`                                                     |
+|                        | `updateAnalysis(article)`                      | `Promise<void>`                                                                  |
+|                        | `findCandidatesForNewsletter()`                | `Promise<ArticleForGenerateContent[]>`                                           |
+| `TagRepository`        | `findAllTags()`                                | `Promise<string[]>`                                                              |
+| `NewsletterRepository` | `getNextIssueOrder()`                          | `Promise<number>`                                                                |
+|                        | `saveNewsletter({ newsletter, usedArticles })` | `Promise<{ id: string \| number }>`                                              |
 
-  // Optional parameters:
-  logger: console,
-  publishDate: '2026-02-20',        // Override publication date (ISO format)
-  templateOptions: { /* ... */ },    // Newsletter template customization
-  customFetch: proxyFetch,           // Custom fetch for proxy-based crawling
-  previewNewsletter: {
-    fetchNewsletterForPreview: async () => db.newsletters.latest(),
-    emailService: resendEmailService,
-    emailMessage: { from: 'news@example.com', to: 'preview@example.com' },
-  },
-});
-```
+Candidate selection belongs to your repository. Persist `usedArticles` associations as needed to exclude previously published content. Issue numbering is initialized before the pipeline starts; coordinate concurrent runs in your application or database. `TaskService` tracks an active task within one service instance, not across processes or separate `generateNewsletter()` calls.
+
+### Optional generation settings
+
+| Option                   | Behavior                                                                                                                                             |
+| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `logger`                 | Core `AppLogger` implementation, such as `console`                                                                                                   |
+| `publishDate`            | A real calendar date in `YYYY-MM-DD` format; invalid values throw. Defaults to the current date in `Asia/Seoul` (KST), regardless of server timezone |
+| `customFetch`            | A `typeof fetch` implementation for crawling and parser API requests, such as a proxy adapter; does not configure LLM requests                       |
+| `templateOptions`        | Default or KRAS newsletter branding and Markdown sections (see below)                                                                                |
+| `promptProvider`         | Core `PromptProvider` overriding the package's own LLM prompts entirely; omit to use the package's prompts, which fall back to core's defaults       |
+| `excavationReportSource` | Supplies 국가유산청 발굴조사 보고서 entries from the application; that board is then never crawled. Omit to crawl it as before                       |
+| `previewNewsletter`      | Fetch a saved `Newsletter` and send it through a supplied core `EmailService`                                                                        |
+
+`previewNewsletter` requires `fetchNewsletterForPreview: () => Promise<Newsletter>`, `emailService` (with `send(message)`), and `emailMessage` (core `EmailMessage` without `subject`, `html`, or `text`). The core fills those three fields and skips preview delivery when no newsletter was created. Ensure the callback fetches the issue saved by this run.
 
-**Repository interfaces** are defined in `src/types/dependencies.ts`. Each method signature includes JSDoc with expected input/output types.
+Production integration must also supply a scheduler, subscriber storage, bulk delivery, and unsubscribe handling. The daily parser health-check workflow does not generate or distribute newsletters.
 
 ## Architecture
 
@@ -136,41 +161,138 @@ const newsletterId = await generateNewsletter({
 
 Uses the **Provider-Service pattern** from `@llm-newsletter-kit/core`. See [core docs](https://github.com/heripo-lab/llm-newsletter-kit-core#architecture--flow) for flow diagrams.
 
-## Components
+## Configuration and models
+
+Defaults below describe the checked-in code, not provider recommendations. This package uses AI SDK 7 and the OpenAI, Anthropic, and Google SDK adapters.
+
+| Stage              | Provider  | Default model            |
+| ------------------ | --------- | ------------------------ |
+| Tag classification | OpenAI    | `gpt-5.6-luna`           |
+| Image analysis     | OpenAI    | `gpt-5.6-terra`          |
+| Importance scoring | OpenAI    | `gpt-5.6-terra`          |
+| Content generation | OpenAI    | `gpt-5.6-sol`            |
+| Content generation | Anthropic | `claude-sonnet-4-6`      |
+| Content generation | Google    | `gemini-3.1-pro-preview` |
+
+Select content generation with `contentGeneration: { provider, apiKey, model? }`. `model` overrides that provider's default. Analysis models are configured in [analysis.provider.ts](./src/providers/analysis.provider.ts).
+
+[src/config/index.ts](./src/config/index.ts) defines Korean output (`outputLanguage: '한국어'`), the cultural heritage domain (`expertField: ['문화유산']`), brand name, `subscribePageUrl`, LLM `maxRetries: 5`, chain `stopAfterAttempt: 3`, and generation `temperature: 0.3`. Publication settings are `minimumArticleCountForIssue: 5` and `priorityArticleScoreThreshold: 8`; the core engine evaluates them. In the locked core 3.0.5 implementation, the count check skips **5 or fewer** candidates unless at least one has importance score >= 8. An empty candidate list is always skipped.
+
+## Crawling targets and parsers
+
+[src/config/crawling-targets.ts](./src/config/crawling-targets.ts) currently defines these board targets (multiple boards may belong to one organization):
+
+| Group      | Active | Commented out |
+| ---------- | -----: | ------------: |
+| News       |     57 |             1 |
+| Business   |      4 |             0 |
+| Employment |     12 |             0 |
+| **Total**  | **73** |         **1** |
+
+Only the excavation status board stays commented out, as low-value fragmented data. Boards that a site's robots.txt restricts are configured normally and refused at runtime by the robots.txt check, so the configuration does not have to track each site's policy by hand. With current policies 14 of the 73 targets are refused.
+
+Two Employment targets are read from data.go.kr open APIs instead of scraped: 나라일터 (`PblJobService`) and 알리오 (`recruitment`). Each is a single request, and `publicDataApiKey` supplies the service key — omit it and both answer with an empty list without making a request. `src/crawling/heritage-job-filter.ts` narrows them before analysis, since the boards carry every public-sector vacancy in the country; roughly 2% survive, about 2.6 postings a day.
+
+Crawling fetches pass through that check (`src/crawling/robots.ts`) before reaching the network: a disallowed request is refused rather than sent, and a missing or unreachable robots.txt allows it. `robotsExemptOrigins` in `src/config/index.ts` lists origins exempted from the check, each with its reason.
+
+Sources include the Korea Heritage Service, National Research Institute of Cultural Heritage, National Research Institute of Maritime Heritage, Korea Heritage Agency, Korea Association of Archaeological Heritage, archaeological societies, and national museums.
+
+[src/parsers/](./src/parsers/) contains 22 organization-specific parser modules plus shared date and URL utilities. List parsers return `ParsedTargetListItem[]` (title, date, detail URL, date type, and optional source ID); detail parsers return Markdown `detailContent` and attachment/image flags. Parsers can be synchronous or asynchronous. KRAS, Yeongnam Archaeological Society, and maritime heritage sources use additional API requests for client-rendered content.
+
+`CrawlingProvider` uses a maximum concurrency of 5 and wraps the supplied fetch to route KRAS public detail URLs to its detail API, while retaining public URLs in article metadata. When constructing your own pipeline, use the provider's fetch together with its target groups.
+
+To display the active sources without crawling:
+
+```typescript
+import { getSourceList } from '@heripo/research-radar';
+
+const groups = getSourceList(); // [{ id, name, sources: [{ id, name, url }] }]
+```
+
+`createCrawlingTargetGroups(customFetch?)`, `getSourceList()`, `contentOptions`, `newsletterConfig`, `llmConfig`, and `researchRadarPromptProvider` are public exports. `ExcavationReport` and `ExcavationReportSource` are exported as types. The package also exports the three provider classes, `DateService`, `TaskService`, and their public configuration/dependency types through [src/index.ts](./src/index.ts).
 
-**Config** (`src/config/`): Brand, language, LLM settings
+## Email templates
 
-**Targets** (`src/config/crawling-targets.ts`): 66 sources (News 52, Business 4, Employment 10)
+[src/templates/](./src/templates/) contains the responsive newsletter template, welcome email template, and shared logos, introduction, sanitization, and footer components. Templates include light/dark mode styles and heripo/KRAS variants. Markdown sections are converted with `safe-markdown2html`; welcome email names are sanitized with DOMPurify and CSS is inlined with `juice`.
 
-**Parsers** (`src/parsers/`): Custom extractors per organization
+`NewsletterTemplateOptions` supports:
+
+| Option                  | Purpose                                                                                                                                            |
+| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `isKrasNewsletter`      | Enable KRAS branding, the 50th-anniversary header, archaeology-first expertise, free-form introduction, and the brand name `한국고고학회 뉴스레터` |
+| `krasNewsMarkdown`      | Society news before generated content                                                                                                              |
+| `heripolabNewsMarkdown` | heripo lab news before generated content                                                                                                           |
+| `krasNoticeMarkdown`    | Society notices after generated content                                                                                                            |
+| `titleContext`          | Title-generation context, only when `isKrasNewsletter: true`; an empty string is ignored                                                           |
+| `displayDate`           | Header date; `generateNewsletter()` replaces it with the date resolved by `DateService` when template options are supplied                         |
+
+The three Markdown sections are available in either branding mode.
+
+```typescript
+import {
+  type NewsletterTemplateOptions,
+  generateWelcomeHTML,
+} from '@heripo/research-radar';
+
+const templateOptions: NewsletterTemplateOptions = {
+  isKrasNewsletter: true,
+  titleContext: '한국고고학전국대회',
+  krasNewsMarkdown: '학회 소식을 입력하세요.',
+  krasNoticeMarkdown: '학회 공지사항을 입력하세요.',
+};
+
+const welcomeHtml = await generateWelcomeHTML('subscriber-id', '홍길동', {
+  isKrasNewsletter: true,
+  siteUrl: 'https://heripo.app',
+});
+```
 
-**Templates** (`src/templates/`): `newsletter-html.ts` (responsive email with light/dark mode), `welcome-html.ts` (`generateWelcomeHTML()` async), `shared.ts` (shared HTML components)
+`generateWelcomeHTML(id, name, options?)` returns `Promise<string>` and only renders HTML. `siteUrl` defaults to `https://heripo.app`; unsubscribe links use `/research-radar/unsubscribe?id=...`. Newsletter templates retain Resend's `{{{RESEND_UNSUBSCRIBE_URL}}}` marker, which your delivery integration must resolve or replace. The raw newsletter template factory and shared HTML helpers are internal, not package-root exports.
 
 ## Development commands
 
 ```bash
+# Install development dependencies from the lockfile
+npm ci
+
 # build
+npm run clean              # remove dist/
 npm run build              # clean dist/ and build with Rollup (ESM + types)
 
 # type-check & lint
 npm run lint               # lint source files
 npm run lint:fix           # lint with autofix
+npm run lint:ci            # quiet CI lint
 npm run typecheck          # TypeScript type-check
 
 # formatting
-npm run format             # Prettier formatting
+npm run format             # format src/ with Prettier
+npm run format:check       # check src/ formatting
 ```
 
+The build emits ESM, declarations, and a JavaScript sourcemap; runtime dependencies remain external. CI runs `npm ci`, `format:check`, `lint:ci`, `typecheck`, and `build` on Node.js 24.x for pull requests and manual dispatch. Formatting scripts cover `src/`; to check the READMEs explicitly, run `npx prettier --check README.md README-ko.md`. There is no `npm test` script.
+
+For maintainers, `release` publishes to npm, while `release:patch`, `release:minor`, and `release:major` bump the version and publish. The version hooks build first and push commits/tags; `prepublishOnly` builds before publishing.
+
 ### Crawler Debugger
 
 A web-based tool for testing crawling parsers during development. Built with Express.js and vanilla HTML/CSS/JS to minimize dependencies.
 
 ```bash
 npm run dev:crawler        # Start at http://localhost:3333
-npm run dev:crawler:proxy  # Start with proxy support (uses .env)
+npm run dev:crawler:proxy  # Load dev-tools/crawler-debugger/.env
+```
+
+For proxy use, create the gitignored `dev-tools/crawler-debugger/.env` file with your proxy endpoint:
+
+```dotenv
+PROXY_URL=http://127.0.0.1:8080
 ```
 
+The library itself receives a `customFetch` function; it does not read `PROXY_URL`.
+
 **Features**:
+
 - Test `parseList()` and `parseDetail()` parsers via web UI
 - View raw HTML source for debugging
 - Copy parsed results as JSON
@@ -185,7 +307,7 @@ Preview rendered newsletter HTML with sample content.
 npm run dev:newsletter-preview  # Start at http://localhost:3334
 ```
 
-Query params: `?kras=true` (KRAS mode), `?krasNews=true` (KRAS news section), `?heripolabNews=true` (heripo lab news section)
+Use the UI controls, or open `/api/preview?kras=true&krasNews=true&krasNotice=true&heripolabNews=true` on port 3334. These query parameters belong to `/api/preview`, not the root UI URL. Each section switch includes sample Markdown; this preview does not call an LLM.
 
 ### Welcome Email Preview
 
@@ -195,7 +317,7 @@ Preview rendered welcome email HTML.
 npm run dev:welcome-preview  # Start at http://localhost:3335
 ```
 
-Query params: `?kras=true` (KRAS mode), `?name=홍길동` (subscriber name)
+Use the UI controls, or open `/api/preview?kras=true&name=홍길동` on port 3335. No API key is needed for either HTML preview tool.
 
 ### Parser Health-Check
 
@@ -203,19 +325,20 @@ CLI tool that validates all active crawling parsers against live websites. Detec
 
 ```bash
 npm run health-check        # Run health-check
-npm run health-check:proxy  # Run with proxy support (uses .env)
+npm run health-check:proxy  # Load the same .env and pass --proxy
 npm run health-check -- --skip-khs-excavation  # Skip KHS excavation report/site-open targets
 ```
 
 **What it checks per target**:
-- `parseList()`: Returns non-empty array with valid title, date, and detailUrl
-- `parseDetail()`: Returns non-empty detailContent (20+ chars)
 
-Use `--skip-target=<id-or-name>` to exclude additional known-flaky targets from a run.
+- `parseList()`: Non-empty array; checks the first item for non-empty title/date and a detailUrl starting with `http`
+- `parseDetail()`: Fetches the first detail item and checks trimmed `detailContent` length >= 20
+
+Use repeatable `--skip-target=<id-or-name>` or `--skip-target <id-or-name>` to exclude targets by exact ID or name. `npm run health-check -- --help` lists options without crawling. This is a live-site smoke check of the first item per target, not full article validation; it does not invoke LLM analysis. The script exits with code 1 if any checked target fails.
 
 **Output**: Console table summary + compact text summary for CI integrations.
 
-**CI**: Daily automated run via GitHub Actions (`.github/workflows/parser-health-check.yml`) with Slack notifications on pass/fail. CI skips the KHS excavation report/site-open targets because their fetches can fail in Actions even when normal crawling works.
+**CI**: [.github/workflows/parser-health-check.yml](./.github/workflows/parser-health-check.yml) runs daily at 08:00 UTC (17:00 KST), or manually, on an `org-linux` runner with a 30-minute job timeout. It skips the two KHS excavation report/site-open targets. The health-check composes the same fetch as production — robots.txt gate, then the KRAS detail adapter — so disallowed boards are reported as skipped rather than failed: 16 skipped and 55 checked with the current configuration. The detail check tries up to three list items, so one unreadable post at the top of a board does not fail the target. Slack notifications require the `SLACK_BOT_TOKEN` secret and `SLACK_ALERT_DEV_CHANNEL` repository variable. Forks need a matching runner and notification configuration to use this workflow unchanged. The CLI also writes GitHub Actions outputs and a job summary when their environment variables are present.
 
 ## 🤝 Contributing
 
@@ -230,55 +353,58 @@ See [CONTRIBUTING.md](./CONTRIBUTING.md) for contribution workflow, dev setup, a
 
 To build your own newsletter, update these files:
 
-**1. Template** (`src/templates/newsletter-html.ts`):
+**1. Templates** (`src/templates/newsletter-html.ts`, `welcome-html.ts`, `shared.ts`):
+
 - Logo URLs, brand colors (#D2691E, #E59866), contact info
 - Platform intro and footer text
 - Unsubscribe link format (currently Resend's `{{{RESEND_UNSUBSCRIBE_URL}}}`)
 
 **2. Config** (`src/config/index.ts`):
+
 ```typescript
-brandName: 'Your Newsletter Name'
-subscribeUrl: 'https://yourdomain.com/subscribe'
+import type { NewsletterConfig } from '@heripo/research-radar';
+
+export const newsletterConfig: NewsletterConfig = {
+  brandName: 'Your Newsletter Name',
+  subscribePageUrl: 'https://yourdomain.com/subscribe',
+  publicationCriteria: {
+    minimumArticleCountForIssue: 5,
+    priorityArticleScoreThreshold: 8,
+  },
+};
 ```
 
 **3. Crawling targets** (`src/config/crawling-targets.ts`):
+
 - Replace Korean heritage sites with your domain sources
 - Implement parsers in `src/parsers/`
 
 **4. Switch content generation LLM provider** (optional):
 
 Content generation supports **3 built-in providers** — just change `contentGeneration.provider`:
+
 ```typescript
-contentGeneration: {
-  provider: 'google',  // 'openai' | 'anthropic' | 'google'
-  apiKey: process.env.GOOGLE_API_KEY,
-  model: 'gemini-3.1-pro-preview',  // optional, each provider has a default
-}
-```
-Default models: openai=`gpt-5.1`, anthropic=`claude-sonnet-4-6`, google=`gemini-3.1-pro-preview`
+import type { ContentGenerationConfig } from '@heripo/research-radar';
 
-Analysis provider (OpenAI) can be changed by modifying `src/providers/analysis.provider.ts`. Any [Vercel AI SDK provider](https://sdk.vercel.ai/providers) works.
+const apiKey = process.env.GOOGLE_API_KEY;
+if (!apiKey) throw new Error('GOOGLE_API_KEY is required');
 
-**Search keywords**: `heripo`, `kimhongyeon`, `#D2691E`, `openai`, `gpt-5`, `contentGeneration`
+const contentGeneration: ContentGenerationConfig = {
+  provider: 'google',
+  apiKey,
+  model: 'gemini-3.1-pro-preview',
+};
+```
 
-## Why Code-Based?
+Default models: openai=`gpt-5.6-sol`, anthropic=`claude-sonnet-4-6`, google=`gemini-3.1-pro-preview`
 
-Code-based automation delivers **superior output quality** through advanced AI techniques:
+To change analysis providers, update both `src/providers/analysis.provider.ts` (provider type and models) and `src/newsletter-generator.ts` (provider construction), using a compatible AI SDK provider. Also adapt domain-specific minimum-score rules in the analysis provider, output language and expert fields in config, package metadata, and the GitHub Actions runner/Slack settings for your fork.
 
-**No-code platforms**: Generic content, limited to built-in features
-**This kit**: Self-reflection, chain-of-thought, multi-step verification workflows
+**Search keywords**: `heripo`, `kimhongyeon`, `#D2691E`, `openai`, `gpt-5`, `contentGeneration`
 
-**Key advantages**:
-- **Quality**: Sophisticated prompting strategies, custom validation pipelines
-- **Cost control**: Different models per step, token limits, retry logic
-- **Flexibility**: Swap any component (Crawling/Analysis/Content/Email) via Provider interfaces
-- **Operations**: Built-in retries, preview emails, integrates with CI/CD
-- **No lock-in**: OSS, self-hostable, any LLM provider
+## Why Code-Based?
 
-**Design philosophy**:
-- Logic in code (orchestration, deduplication)
-- Reasoning in AI (analysis, scoring, content generation)
-- Connections in architecture (swappable Providers)
+The domain logic is inspectable in source: parser behavior, model selection, score rules, publication settings, and HTML templates can be versioned together. Repository and provider interfaces let applications integrate existing storage and delivery systems, while the core engine supplies orchestration and retry handling.
 
 ## Related Projects
 

package/dist/index.d.ts

@@ -1,4 +1,4 @@
-import { UrlString, ParsedTarget, CrawlingTargetGroup, CrawlingTarget, UnscoredArticle, ArticleForUpdateByAnalysis, ArticleForGenerateContent, Newsletter, AppLogger, EmailService, EmailMessage, DateService as DateService$1, IsoDateString, TaskService as TaskService$1, AnalysisProvider as AnalysisProvider$1, ContentGenerateProvider as ContentGenerateProvider$1, HtmlTemplate, CrawlingProvider as CrawlingProvider$1, GenerateNewsletterConfig } from '@llm-newsletter-kit/core';
+import { UrlString, ParsedTarget, CrawlingTargetGroup, CrawlingTarget, UnscoredArticle, ArticleForUpdateByAnalysis, ArticleForGenerateContent, Newsletter, IsoDateString, AppLogger, EmailService, EmailMessage, PromptProvider, DateService as DateService$1, TaskService as TaskService$1, AnalysisProvider as AnalysisProvider$1, ContentGenerateProvider as ContentGenerateProvider$1, HtmlTemplate, CrawlingProvider as CrawlingProvider$1, GenerateNewsletterConfig } from '@llm-newsletter-kit/core';
 import { OpenAIProvider } from '@ai-sdk/openai';
 import { LanguageModel } from 'ai';
 
@@ -54,6 +54,53 @@ interface ArticleRepository {
      */
     findCandidatesForNewsletter(): Promise<ArticleForGenerateContent[]>;
 }
+/**
+ * A single excavation report supplied by the application instead of crawled.
+ *
+ * Field names mirror what the 국가유산청 report board exposes, so an application
+ * that already stores these rows can hand them over without reshaping them.
+ */
+interface ExcavationReport {
+    /**
+     * The board's own identifier (`ecexmRcno`). Used both as the article's unique
+     * id and to build its public detail URL, so it must match the value the board
+     * uses — otherwise previously crawled reports are re-saved as duplicates.
+     * @example "202609157717"
+     */
+    externalId: string;
+    /**
+     * Report title (보고서명).
+     * @example "태안 태안읍성 -남문지 및 연지-"
+     */
+    title: string;
+    /**
+     * Submission date (제출일) in ISO format (YYYY-MM-DD).
+     * @example "2026-09-15"
+     */
+    submittedDate: IsoDateString;
+    /**
+     * Report detail fields as label/value pairs, rendered into the article body in
+     * insertion order. Supply whatever the source holds — 허가번호, 유적명,
+     * 발간기관, 조사시도시군구, 조사기간, 유적성격/시대구분 and so on.
+     * Empty values are skipped.
+     */
+    fields: Record<string, string | null | undefined>;
+    /**
+     * Whether the report has a downloadable file. Defaults to true, matching the
+     * crawled parser.
+     */
+    hasAttachedFile?: boolean;
+}
+/**
+ * Supplies excavation reports from the application instead of crawling them.
+ *
+ * When provided, the 국가유산청 발굴조사 보고서 board is served from this
+ * function and never requested over the network; the rest of the crawl is
+ * unaffected. When omitted, the board is crawled as before.
+ *
+ * Called at most once per generation run.
+ */
+type ExcavationReportSource = () => Promise<ExcavationReport[]>;
 /**
  * Repository interface for tag management
  */
@@ -178,7 +225,7 @@ interface PreviewNewsletterOptions {
  * Each provider uses a sensible default model that can be overridden.
  *
  * Default models:
- * - openai: `gpt-5.4`
+ * - openai: `gpt-5.6-sol`
  * - anthropic: `claude-sonnet-4-6`
  * - google: `gemini-3.1-pro-preview`
  */
@@ -226,6 +273,30 @@ interface NewsletterGeneratorDependencies {
     templateOptions?: NewsletterTemplateOptions;
     /** Custom fetch function for crawling (e.g., proxy-based fetch). Optional. */
     customFetch?: typeof fetch;
+    /**
+     * Supplies 국가유산청 발굴조사 보고서 entries from the application (optional).
+     *
+     * When provided, that board is read from this function and never crawled,
+     * which lets an application that already stores the reports reuse them. Omit
+     * it to keep crawling the board as before. No other target is affected.
+     */
+    excavationReportSource?: ExcavationReportSource;
+    /**
+     * data.go.kr service key for the 나라일터 and 알리오 job boards (optional).
+     *
+     * Pass the encoded key exactly as the portal supplies it. Both boards are read
+     * through open APIs rather than scraped; omit the key and they collect
+     * nothing while every other target is unaffected.
+     */
+    publicDataApiKey?: string;
+    /**
+     * LLM prompt overrides (optional).
+     *
+     * When provided, this replaces Research Radar's own prompt provider entirely
+     * rather than merging with it. Omit it to use the package's tuned prompts,
+     * which in turn fall back to core's defaults for any stage they do not define.
+     */
+    promptProvider?: PromptProvider;
 }
 /**
  * Newsletter generation execution function
@@ -433,7 +504,7 @@ declare class CrawlingProvider implements CrawlingProvider$1 {
     customFetch?: typeof fetch;
     /** Crawling target groups configuration */
     crawlingTargetGroups: CrawlingTargetGroup[];
-    constructor(articleRepository: ArticleRepository, customFetch?: typeof fetch);
+    constructor(articleRepository: ArticleRepository, customFetch?: typeof fetch, excavationReportSource?: ExcavationReportSource, logger?: AppLogger, publicDataApiKey?: string);
     /**
      * Fetch existing articles by URLs to avoid duplicate crawling
      * @param articleUrls - URLs to check
@@ -489,6 +560,38 @@ declare const contentOptions: ContentOptions;
  * Newsletter brand configuration
  */
 declare const newsletterConfig: NewsletterConfig;
+/**
+ * Maximum importance score per heritage domain (`tag1`).
+ *
+ * The newsletter is archaeology-first: archaeology and cultural heritage keep
+ * the full 1-10 range, while natural and intangible heritage are capped so they
+ * cannot crowd out archaeological coverage. `기타` — material that is not
+ * heritage at all — is capped lowest, because the scoring prompt's academic-value
+ * floor is domain-blind and would otherwise lift things like a general journal's
+ * call for papers into the top half of the scale.
+ *
+ * The cap is applied deterministically after scoring, in
+ * `AnalysisProvider.update()`, rather than asked for in the prompt, so the
+ * ceiling always holds. It never reaches 1: a score of 1 means "exclude from the
+ * newsletter" in the consuming application's candidate query, so capping to 1
+ * would delete these articles instead of demoting them.
+ *
+ * Domains absent from this map are not capped.
+ */
+declare const maximumImportanceScoreByDomain: Record<string, number>;
+/**
+ * Origins exempted from the robots.txt check.
+ *
+ * Each entry deliberately overrides what the site publishes, so it needs a
+ * reason and should be revisited when that site's robots.txt changes.
+ *
+ * - `http://www.yngogo.or.kr` (영남고고학회): its board is rendered from
+ *   `/module/ntt/unity/selectNttListAjax.ink`, and robots.txt carries a blanket
+ *   `Disallow: /module`. The rule reads as protecting an internal path rather
+ *   than the public board it happens to serve, and there is no other route to
+ *   the listing, so the society's boards are collected under this exemption.
+ */
+declare const robotsExemptOrigins: readonly string[];
 /**
  * LLM configuration
  */
@@ -500,5 +603,35 @@ declare const llmConfig: {
     };
 };
 
-export { AnalysisProvider, ContentGenerateProvider, CrawlingProvider, DateService, TaskService, contentOptions, createCrawlingTargetGroups, generateNewsletter, generateWelcomeHTML, getSourceList, llmConfig, newsletterConfig };
-export type { ArticleRepository, ContentGenerationConfig, ContentOptions, NewsletterConfig, NewsletterGeneratorDependencies, NewsletterRepository, NewsletterTemplateOptions, PreviewNewsletterOptions, SourceGroup, SourceItem, TagRepository, TaskRepository, WelcomeTemplateOptions };
+/**
+ * Research Radar's LLM prompt overrides.
+ *
+ * Each builder here **replaces** core's built-in prompt for that stage rather
+ * than extending it — a `PromptBuilder` returns the whole prompt string. That is
+ * deliberate: core's defaults instruct the model to use emoticons, render
+ * importance as star ratings, and add share-of-coverage statistics, all of which
+ * this newsletter removes. Appending contradicting rules to those defaults
+ * degrades the output, so a replacement is written from scratch instead.
+ *
+ * **Contract a replacement must still satisfy.** Core's output schema is fixed,
+ * and `generateNewsletter` regenerates the whole newsletter whenever the model
+ * reports a failure. A replacement prompt has to steer the model toward:
+ *
+ * - `title`: 20–70 characters
+ * - `isWrittenInOutputLanguage`, `copyrightVerified`, `factAccuracy`: true
+ * - `titleContext` (KRAS mode): the phrase must appear in the title
+ *
+ * Core caps that loop at 5 attempts, so a prompt that ignores the contract costs
+ * up to 5 full generations on the most expensive model in the pipeline.
+ *
+ * Tune these against real articles with core's playground
+ * (`npm run playground:generate-newsletter` in ../llm-newsletter-kit-core) and
+ * diff them against the defaults for free with `playground:verify-prompts`.
+ * The playground loads this module from `dist`, so run `npm run build` here
+ * after every edit — its source cannot be imported directly across repos
+ * because of the `~/*` path alias.
+ */
+declare const researchRadarPromptProvider: PromptProvider;
+
+export { AnalysisProvider, ContentGenerateProvider, CrawlingProvider, DateService, TaskService, contentOptions, createCrawlingTargetGroups, generateNewsletter, generateWelcomeHTML, getSourceList, llmConfig, maximumImportanceScoreByDomain, newsletterConfig, researchRadarPromptProvider, robotsExemptOrigins };
+export type { ArticleRepository, ContentGenerationConfig, ContentOptions, ExcavationReport, ExcavationReportSource, NewsletterConfig, NewsletterGeneratorDependencies, NewsletterRepository, NewsletterTemplateOptions, PreviewNewsletterOptions, SourceGroup, SourceItem, TagRepository, TaskRepository, WelcomeTemplateOptions };