greptor 0.2.1 → 0.4.0

package/README.md

@@ -34,22 +34,23 @@ bun add greptor
 
 ### Step 2: Initialize
 
-Create a Greptor instance with your base directory, topic, and LLM model.
+Create a Greptor instance with your base directory, topic, and AI SDK language model.
 
 ```typescript
 import { createGreptor } from 'greptor';
+import { openai } from "@ai-sdk/openai";
 
 // Create Greptor instance
 const greptor = await createGreptor({
   baseDir: './projects/investing',
   topic: 'Investing, stock market, financial, and macroeconomics',
-  llmModel: 'openai:gpt-5-mini'
+  model: openai("gpt-5-mini"),
 });
 ```
 
 - **`baseDir`**: Home directory for your project where all data will be stored.
-- **`topic`**: Helps Greptor understand your data better and generate a relevant metadata schema.
-- **`llmModel`**: OpenAI-compatible model for chunking and metadata extraction. You must provide an API key via environment variables.
+- **`topic`**: Helps Greptor understand your data better and generate a relevant tag schema.
+- **`model`**: A `LanguageModel` instance from [Vercel AI SDK](https://ai-sdk.dev).
 
 ### Step 3: Start Feeding Documents
 
@@ -62,8 +63,8 @@ await greptor.eat({
   label: 'Top Five AI Stocks I\'m Buying Now',
   content: '{fetch and populate video transcript here}',
   creationDate: new Date('2025-11-15'),
-  metadata: {
-    // Optional custom metadata specific to the source or document
+  tags: {
+    // Optional custom tags specific to the source or document
     channelTitle: 'Joseph Carlson',
     channelSubscribers: 496000
   },
@@ -77,8 +78,8 @@ await greptor.eat({
   label: 'Tesla reports 418,227 deliveries for the fourth quarter, down 16%',
   content: '{fetch and populate Reddit post with comments here}',
   creationDate: new Date('2025-12-03'),
-  metadata: {
-    // Optional custom metadata
+  tags: {
+    // Optional custom tags
     upvotes: 1400
   },
 });
@@ -86,7 +87,7 @@ await greptor.eat({
 
 ### Step 4: Wait for Background Processing
 
-Greptor will write your input to a raw Markdown file immediately, then run background enrichment (LLM cleaning + chunking + metadata extraction) and write a processed Markdown file. You can grep the raw files right away, and the processed files will appear shortly after.
+Greptor will write your input to a raw Markdown file immediately, then run background enrichment (LLM cleaning + chunking + tagging) and write a processed Markdown file. You can grep the raw files right away, and the processed files will appear shortly after.
 
 ### Step 5: Generate a Claude Code Skill
 
@@ -96,7 +97,7 @@ await greptor.createSkill(['youtube', 'reddit']);
 
 This generates a Claude Code skill that instructs agents on how to search your indexed content effectively.
 
-The skill is customized for the sources you provide and includes search tips based on the metadata schema. You can always customize it manually further for better results.
+The skill is customized for the sources you provide and includes search tips based on the tag schema. You can always customize it manually further for better results.
 
 ### Step 6: Run the Agent
 
@@ -150,9 +151,7 @@ Now you have a personal investment research assistant with access to your portfo
 
 Workers pick up new documents and run a one-time pipeline:
 
-1. **LLM cleaning**: Remove timestamps, ads, disclaimers, boilerplate, and irrelevant content.
-2. **LLM chunking**: Transform a blob into semantic section chunks.
-3. **LLM metadata extraction**: Extract metadata relevant to your topic/domain and enrich each chunk with denormalized metadata.
+1. **LLM clean + chunk + tag (single prompt)**: Remove boilerplate, split into semantic chunks, and inline grep-friendly per-chunk tags.
 
 Here's an example of a processed file:
 
@@ -165,121 +164,142 @@ date: 2025-11-15
 ticker: "NVDA"
 videoId: "dQw4w9WgXcQ"
 url: "https://youtube.com/watch?v=dQw4w9WgXcQ"
-chunks:
-  - id: c01
-    title: "Revenue Growth Analysis"
-    topics: [earnings, revenue, data-center]
-    sentiment: positive
-    tickers: [NVDA]
-    price_mentioned_usd: 850.50
-    revenue_mentioned_billions: 35.1
-  - id: c02
-    title: "AI Chip Demand Outlook"
-    topics: [ai, competition, market-share]
-    sentiment: bullish
-    tickers: [NVDA, AMD, INTC]
-    timeframe: next-quarter
 ---
 
-CHUNK c01: "Revenue Growth Analysis"
+## 01 Revenue Growth Analysis
+topics=earnings,revenue,data_center
+sentiment=positive
+tickers=NVDA
+
 NVIDIA reported Q4 revenue of $35.1 billion, beating estimates...
 
-CHUNK c02: "AI Chip Demand Outlook"
+## 02 AI Chip Demand Outlook
+topics=ai,competition,market_share
+sentiment=bullish
+tickers=NVDA,AMD,INTC
+timeframe=next_quarter
+
 The demand for AI accelerators continues to outpace supply...
 ```
 
 ### 3) Navigate with grep/glob
 
-Your "index" is the YAML frontmatter combined with the file layout. Agents can search it deterministically.
+Your "index" is the YAML frontmatter (document-level) plus the per-chunk tag lines. Agents can search it deterministically.
 
-**Search examples**:
+**Basic search examples**:
 
 ```bash
-# Find all bullish sentiment for TSLA stock
-rg -l "ticker:.*TSLA" content/processed | xargs rg "sentiment:.*bullish"
+# Simple tag search with context
+rg -n -C 6 "ticker=NVDA" content/processed/
 
-# Count documents per ticker
-rg "ticker:" content/processed -o | sort | uniq -c | sort -rn | head -20
+# Search for any value in a tag field
+rg -n -C 6 "sentiment=" content/processed/
 
-# What companies does a specific YouTuber discuss?
-rg "company:" content/processed/youtube/JosephCarlsonShow -o | sort | uniq -c | sort -rn
+# Case-insensitive full-text search
+rg -i -n -C 3 "artificial intelligence" content/processed/
+
+# Search within a specific source
+rg -n -C 6 "sector=technology" content/processed/youtube/
+```
+
+**Date-filtered searches**:
+
+```bash
+# Content from December 2025
+rg -n -C 6 "ticker=TSLA" content/processed/ --glob "**/2025-12/*.md"
 
-# Find all AI-related narratives with strong buy recommendations
-rg -l "narrative:.*ai_boom" content/processed | xargs rg "recommendation:.*strong_buy"
+# Q4 2025 content
+rg -n -C 6 "sentiment=bullish" content/processed/ --glob "**/2025-1[0-2]/*.md"
 
-# Technology sector stocks with bullish sentiment in December 2025
-rg -l "sector:.*technology" content/processed --glob "**/2025-12/*.md" | xargs rg "sentiment:.*bullish"
+# Specific month and source
+rg -n -C 6 "asset_type=etf" content/processed/reddit/ --glob "**/2025-11/*.md"
+```
 
-# Find dividend investment style discussions
-rg "investment_style:.*dividend" content/processed -l | head -10
+**Combined tag filters**:
 
-# Bearish sentiment on large-cap stocks
-rg -l "market_cap:.*large_cap" content/processed | xargs rg "sentiment:.*bearish"
+```bash
+# Match chunks with two specific tags (using file list)
+rg -l "sector=technology" content/processed/ | xargs rg -n -C 6 "sentiment=bullish"
 
-# List all tickers mentioned with their sentiment
-rg "ticker: \[.*\]" content/processed -A 5 | rg "sentiment:"
+# Pipeline filter for complex queries
+rg -n -C 6 "ticker=AAPL" content/processed/ | rg "recommendation=.*buy"
 
-# Find EV-related discussions across all sources
-rg "narrative:.*ev_transition" content/processed
+# Three-way filter: tech stocks with bullish sentiment and buy recommendation
+rg -l "sector=technology" content/processed/ | xargs rg -l "sentiment=bullish" | xargs rg -n -C 6 "recommendation=buy"
 
-# Combine multiple filters: tech stocks with strong buy in specific timeframe
-rg -l "sector:.*technology" content/processed --glob "**/2025-11/*.md" | \
-  xargs rg -l "recommendation:.*strong_buy" | \
-  xargs rg "ticker:" -o | sort | uniq -c
+# Find AI narrative discussions with specific tickers
+rg -n -C 6 "narrative=.*ai" content/processed/ | rg "ticker=NVDA\|ticker=.*,NVDA"
+```
+
+**Discovery and exploration**:
+
+```bash
+# List all unique tickers mentioned
+rg -o "ticker=[^\n]+" content/processed/ | cut -d= -f2 | tr ',' '\n' | sort -u
+
+# Count occurrences of each sentiment
+rg -o "sentiment=[^\n]+" content/processed/ | cut -d= -f2 | sort | uniq -c | sort -rn
+
+# Top 20 most discussed companies
+rg -o "company=[^\n]+" content/processed/ | cut -d= -f2 | tr ',' '\n' | sort | uniq -c | sort -rn | head -20
+
+# Find all files discussing dividend investing
+rg -l "investment_style=dividend" content/processed/
+
+# See what narratives exist in the data
+rg -o "narrative=[^\n]+" content/processed/ | cut -d= -f2 | tr ',' '\n' | sort -u
 ```
 
 **Analysis patterns**:
 
 ```bash
-# Aggregate sentiment distribution
-rg "sentiment:" content/processed -o | cut -d: -f2 | tr -d ' ' | sort | uniq -c
+# Sentiment distribution for a specific ticker
+rg -n -C 6 "ticker=TSLA" content/processed/ | rg -o "sentiment=[^\n]+" | cut -d= -f2 | sort | uniq -c
 
 # Most discussed sectors
-rg "sector:" content/processed -o | sort | uniq -c | sort -rn
+rg -o "sector=[^\n]+" content/processed/ | cut -d= -f2 | tr ',' '\n' | sort | uniq -c | sort -rn
 
 # Track narrative evolution over time
 for month in 2025-{10..12}; do
   echo "=== $month ==="
-  rg "narrative:" content/processed --glob "**/$month-*/*.md" -o | sort | uniq -c | sort -rn | head -5
+  rg -o "narrative=[^\n]+" content/processed/ --glob "**/$month/*.md" | cut -d= -f2 | tr ',' '\n' | sort | uniq -c | sort -rn | head -5
 done
 
-# Compare sentiment on specific stock across sources
+# Compare sentiment across sources for a stock
 for source in youtube reddit; do
   echo "=== $source ==="
-  rg -l "ticker:.*AAPL" content/processed/$source | xargs rg "sentiment:" -o | sort | uniq -c
+  rg -n -C 6 "ticker=AAPL" content/processed/$source/ | rg -o "sentiment=[^\n]+" | cut -d= -f2 | tr ',' '\n' | sort | uniq -c
 done
-```
 
-## Configuration
+# Find all strong buy recommendations by sector
+for sector in technology healthcare financials; do
+  echo "=== $sector ==="
+  rg -l "sector=$sector" content/processed/ | xargs rg -n -C 3 "recommendation=strong_buy" | head -5
+done
+```
 
-### LLM Model Format
+**Advanced multi-criteria searches**:
 
-Greptor uses the following LLM model format: `provider:model-name`
+```bash
+# Large-cap tech stocks with bullish sentiment
+rg -l "market_cap=large_cap" content/processed/ | xargs rg -l "sector=technology" | xargs rg -n -C 6 "sentiment=bullish"
 
-The provider is an OpenAI API-compatible provider, such as `openai`, `azure`, `ollama`, `deepseek`, etc.
+# Growth investing discussions about mega-cap stocks
+rg -n -C 6 "investment_style=growth" content/processed/ | rg "market_cap=mega_cap"
 
-**Examples**:
+# ETF recommendations from specific time period
+rg -n -C 6 "asset_type=etf" content/processed/ --glob "**/2025-12/*.md" | rg "recommendation=buy\|recommendation=strong_buy"
 
-```typescript
-llmModel: 'openai:gpt-5-mini'
-llmModel: 'ollama:llama3-70b'
+# Bearish sentiment on specific narrative
+rg -n -C 6 "narrative=ev_transition" content/processed/ | rg "sentiment=bearish"
 ```
 
-**Important**: Use a model at least at the level of GPT-5-mini or better.
-**Required Environment Variables**:
-
-```bash
-# For OpenAI models
-OPENAI_API_KEY=your_key_here
+## Configuration
 
-# For Azure OpenAI models
-AZURE_API_KEY=your_key_here
-AZURE_API_BASE_URL=https://your-azure-endpoint.com/v1
-```
 
-## Metadata Schemas
+## Tag Schemas
 
-If you don't provide a schema, Greptor can initialize one for your topic. However, for better results, provide a custom schema.
+If you don't provide a schema, Greptor can initialize one for your topic. However, for better results, provide a custom tag schema.
 
 Here's a comprehensive example for investment research:
 
@@ -287,8 +307,8 @@ Here's a comprehensive example for investment research:
 const greptor = await createGreptor({
   baseDir: './projects/investing',
   topic: 'Investing, stock market, financial, and macroeconomics',
-  llmModel: 'openai:gpt-5-mini',
-  metadataSchema: [
+  model: openai("gpt-5-mini"),
+  tagSchema: [
     {
       name: 'company',
       type: 'string[]',
@@ -362,4 +382,4 @@ const greptor = await createGreptor({
 
 ## License
 
-MIT © Sergii Vashchyshchuk
+MIT © Sergii Vashchyshchuk

package/dist/greptor.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"greptor.d.ts","sourceRoot":"","sources":["../src/greptor.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EACX,iBAAiB,EACjB,eAAe,EACf,gBAAgB,EAChB,cAAc,EACd,MAAM,YAAY,CAAC;AAcpB,MAAM,WAAW,OAAO;IACvB,GAAG,EAAE,CAAC,KAAK,EAAE,eAAe,KAAK,OAAO,CAAC,gBAAgB,CAAC,CAAC;IAC3D,WAAW,EAAE,CACZ,OAAO,EAAE,MAAM,EAAE,EACjB,SAAS,EAAE,OAAO,KACd,OAAO,CAAC,iBAAiB,CAAC,CAAC;CAChC;AAED,wBAAsB,aAAa,CAAC,OAAO,EAAE,cAAc,GAAG,OAAO,CAAC,OAAO,CAAC,CAqH7E"}
+{"version":3,"file":"greptor.d.ts","sourceRoot":"","sources":["../src/greptor.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EACX,iBAAiB,EACjB,eAAe,EACf,gBAAgB,EAChB,cAAc,EACd,MAAM,YAAY,CAAC;AAapB,MAAM,WAAW,OAAO;IACvB,GAAG,EAAE,CAAC,KAAK,EAAE,eAAe,KAAK,OAAO,CAAC,gBAAgB,CAAC,CAAC;IAC3D,WAAW,EAAE,CACZ,OAAO,EAAE,MAAM,EAAE,EACjB,SAAS,EAAE,OAAO,KACd,OAAO,CAAC,iBAAiB,CAAC,CAAC;CAChC;AAED,wBAAsB,aAAa,CAAC,OAAO,EAAE,cAAc,GAAG,OAAO,CAAC,OAAO,CAAC,CAoH7E"}

package/dist/greptor.js

@@ -1,27 +1,25 @@
 import path from "node:path";
 import YAML from "yaml";
-import { createLlmClient } from "./llm/llm-factory.js";
-import { initializeMetadataSchema } from "./metadata-schema/initialize.js";
+import { initializeTagSchema } from "./tag-schema/initialize.js";
 import { createProcessingQueue, enqueueUnprocessedDocuments, startBackgroundWorkers, } from "./processing/processor.js";
 import { generateSkill } from "./skills/skill-generator.js";
 import { createFileStorage } from "./storage/file-storage.js";
 export async function createGreptor(options) {
-    const { baseDir, logger } = options;
+    const { baseDir, logger, model } = options;
     const contentPath = path.join(baseDir, "content");
     const storage = createFileStorage(contentPath);
     logger?.debug?.("Initializing Greptor", { baseDir, topic: options.topic });
-    const metadataSchema = await initializeMetadataSchema(storage.baseDir, options.llmModel, options.topic, options.metadataSchema, logger);
+    const tagSchema = await initializeTagSchema(storage.baseDir, model, options.topic, options.tagSchema, logger);
     const queue = createProcessingQueue();
     const queuedCount = await enqueueUnprocessedDocuments({
         storage,
         queue,
         ...(logger ? { logger } : {}),
     });
-    const llm = createLlmClient(options.llmModel);
     const ctx = {
         domain: options.topic,
-        metadataSchema: YAML.stringify(metadataSchema),
-        llm,
+        tagSchema: YAML.stringify(tagSchema),
+        model,
         storage,
         ...(logger ? { logger } : {}),
     };
@@ -72,7 +70,7 @@ export async function createGreptor(options) {
                 domain: options.topic,
                 sources,
                 baseDir: options.baseDir,
-                metadataSchema,
+                tagSchema,
                 overwrite,
             }, storage);
             return {

package/dist/greptor.js.map

@@ -1 +1 @@
-{"version":3,"file":"greptor.js","sourceRoot":"","sources":["../src/greptor.ts"],"names":[],"mappings":"AAOA,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,IAAI,MAAM,MAAM,CAAC;AACxB,OAAO,EAAE,eAAe,EAAE,MAAM,sBAAsB,CAAC;AACvD,OAAO,EAAE,wBAAwB,EAAE,MAAM,iCAAiC,CAAC;AAC3E,OAAO,EACN,qBAAqB,EACrB,2BAA2B,EAC3B,sBAAsB,GACtB,MAAM,2BAA2B,CAAC;AACnC,OAAO,EAAE,aAAa,EAAE,MAAM,6BAA6B,CAAC;AAC5D,OAAO,EAAE,iBAAiB,EAAE,MAAM,2BAA2B,CAAC;AAU9D,MAAM,CAAC,KAAK,UAAU,aAAa,CAAC,OAAuB;IAC1D,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC;IACpC,MAAM,WAAW,GAAG,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,SAAS,CAAC,CAAC;IAClD,MAAM,OAAO,GAAG,iBAAiB,CAAC,WAAW,CAAC,CAAC;IAE/C,MAAM,EAAE,KAAK,EAAE,CAAC,sBAAsB,EAAE,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC;IAE3E,MAAM,cAAc,GAAG,MAAM,wBAAwB,CACpD,OAAO,CAAC,OAAO,EACf,OAAO,CAAC,QAAQ,EAChB,OAAO,CAAC,KAAK,EACb,OAAO,CAAC,cAAc,EACtB,MAAM,CACN,CAAC;IAEF,MAAM,KAAK,GAAG,qBAAqB,EAAE,CAAC;IACtC,MAAM,WAAW,GAAG,MAAM,2BAA2B,CAAC;QACrD,OAAO;QACP,KAAK;QACL,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KAC7B,CAAC,CAAC;IAEH,MAAM,GAAG,GAAG,eAAe,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;IAC9C,MAAM,GAAG,GAAG;QACX,MAAM,EAAE,OAAO,CAAC,KAAK;QACrB,cAAc,EAAE,IAAI,CAAC,SAAS,CAAC,cAAc,CAAC;QAC9C,GAAG;QACH,OAAO;QACP,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KAC7B,CAAC;IAEF,sBAAsB,CAAC,EAAE,GAAG,EAAE,KAAK,EAAE,WAAW,EAAE,OAAO,CAAC,OAAO,IAAI,CAAC,EAAE,CAAC,CAAC;IAE1E,MAAM,EAAE,IAAI,EAAE,CAAC,qBAAqB,EAAE;QACrC,KAAK,EAAE,OAAO,CAAC,KAAK;QACpB,MAAM,EAAE,WAAW;KACnB,CAAC,CAAC;IAEH,KAAK,UAAU,GAAG,CAAC,KAAsB;QACxC,IAAI,KAAK,CAAC,MAAM,KAAK,MAAM,EAAE,CAAC;YAC7B,MAAM,EAAE,IAAI,EAAE,CAAC,oBAAoB,EAAE,EAAE,MAAM,EAAE,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC;YAC/D,OAAO;gBACN,OAAO,EAAE,KAAK;gBACd,OAAO,EAAE,uBAAuB,KAAK,CAAC,MAAM,EAAE;aAC9C,CAAC;QACH,CAAC;QAED,MAAM,GAAG,GAAG,MAAM,OAAO,CAAC,cAAc,CAAC,KAAK,CAAC,CAAC;QAEhD,IAAI,GAAG,CAAC,IAAI,KAAK,WAAW,EAAE,CAAC;YAC9B,MAAM,EAAE,IAAI,EAAE,CAAC,mCAAmC,EAAE;gBACnD,GAAG,EAAE,GAAG,CAAC,GAAG;gBACZ,KAAK,EAAE,KAAK,CAAC,KAAK;aAClB,CAAC,CAAC;YACH,OAAO;gBACN,OAAO,EAAE,KAAK;gBACd,OAAO,EAAE,0BAA0B;aACnC,CAAC;QACH,CAAC;QAED,IAAI,GAAG,CAAC,IAAI,KAAK,OAAO,EAAE,CAAC;YAC1B,OAAO;gBACN,OAAO,EAAE,KAAK;gBACd,OAAO,EAAE,GAAG,CAAC,OAAO;aACpB,CAAC;QACH,CAAC;QAED,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACvB,MAAM,EAAE,IAAI,EAAE,CAAC,mBAAmB,EAAE,EAAE,GAAG,EAAE,GAAG,CAAC,GAAG,EAAE,KAAK,EAAE,KAAK,CAAC,KAAK,EAAE,CAAC,CAAC;QAE1E,OAAO;YACN,OAAO,EAAE,IAAI;YACb,OAAO,EAAE,gBAAgB;YACzB,GAAG,EAAE,GAAG,CAAC,GAAG;SACZ,CAAC;IACH,CAAC;IAED,KAAK,UAAU,WAAW,CACzB,OAAiB,EACjB,SAAS,GAAG,KAAK;QAEjB,IAAI,CAAC;YACJ,MAAM,EAAE,IAAI,EAAE,CAAC,8BAA8B,EAAE;gBAC9C,MAAM,EAAE,OAAO,CAAC,KAAK;aACrB,CAAC,CAAC;YAEH,MAAM,EAAE,SAAS,EAAE,GAAG,MAAM,aAAa,CACxC;gBACC,MAAM,EAAE,OAAO,CAAC,KAAK;gBACrB,OAAO;gBACP,OAAO,EAAE,OAAO,CAAC,OAAO;gBACxB,cAAc;gBACd,SAAS;aACT,EACD,OAAO,CACP,CAAC;YAEF,OAAO;gBACN,OAAO,EAAE,IAAI;gBACb,OAAO,EAAE,oBAAoB,SAAS,EAAE;gBACxC,SAAS;aACT,CAAC;QACH,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YAChB,MAAM,YAAY,GACjB,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;YACxD,MAAM,EAAE,KAAK,EAAE,CAAC,6BAA6B,YAAY,EAAE,CAAC,CAAC;YAC7D,OAAO;gBACN,OAAO,EAAE,KAAK;gBACd,OAAO,EAAE,YAAY;aACrB,CAAC;QACH,CAAC;IACF,CAAC;IAED,OAAO;QACN,GAAG;QACH,WAAW;KACX,CAAC;AACH,CAAC"}
+{"version":3,"file":"greptor.js","sourceRoot":"","sources":["../src/greptor.ts"],"names":[],"mappings":"AAOA,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,IAAI,MAAM,MAAM,CAAC;AACxB,OAAO,EAAE,mBAAmB,EAAE,MAAM,4BAA4B,CAAC;AACjE,OAAO,EACN,qBAAqB,EACrB,2BAA2B,EAC3B,sBAAsB,GACtB,MAAM,2BAA2B,CAAC;AACnC,OAAO,EAAE,aAAa,EAAE,MAAM,6BAA6B,CAAC;AAC5D,OAAO,EAAE,iBAAiB,EAAE,MAAM,2BAA2B,CAAC;AAU9D,MAAM,CAAC,KAAK,UAAU,aAAa,CAAC,OAAuB;IAC1D,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,GAAG,OAAO,CAAC;IAC3C,MAAM,WAAW,GAAG,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,SAAS,CAAC,CAAC;IAClD,MAAM,OAAO,GAAG,iBAAiB,CAAC,WAAW,CAAC,CAAC;IAE/C,MAAM,EAAE,KAAK,EAAE,CAAC,sBAAsB,EAAE,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC;IAE3E,MAAM,SAAS,GAAG,MAAM,mBAAmB,CAC1C,OAAO,CAAC,OAAO,EACf,KAAK,EACL,OAAO,CAAC,KAAK,EACb,OAAO,CAAC,SAAS,EACjB,MAAM,CACN,CAAC;IAEF,MAAM,KAAK,GAAG,qBAAqB,EAAE,CAAC;IACtC,MAAM,WAAW,GAAG,MAAM,2BAA2B,CAAC;QACrD,OAAO;QACP,KAAK;QACL,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KAC7B,CAAC,CAAC;IAEH,MAAM,GAAG,GAAG;QACX,MAAM,EAAE,OAAO,CAAC,KAAK;QACrB,SAAS,EAAE,IAAI,CAAC,SAAS,CAAC,SAAS,CAAC;QACpC,KAAK;QACL,OAAO;QACP,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KAC7B,CAAC;IAEF,sBAAsB,CAAC,EAAE,GAAG,EAAE,KAAK,EAAE,WAAW,EAAE,OAAO,CAAC,OAAO,IAAI,CAAC,EAAE,CAAC,CAAC;IAE1E,MAAM,EAAE,IAAI,EAAE,CAAC,qBAAqB,EAAE;QACrC,KAAK,EAAE,OAAO,CAAC,KAAK;QACpB,MAAM,EAAE,WAAW;KACnB,CAAC,CAAC;IAEH,KAAK,UAAU,GAAG,CAAC,KAAsB;QACxC,IAAI,KAAK,CAAC,MAAM,KAAK,MAAM,EAAE,CAAC;YAC7B,MAAM,EAAE,IAAI,EAAE,CAAC,oBAAoB,EAAE,EAAE,MAAM,EAAE,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC;YAC/D,OAAO;gBACN,OAAO,EAAE,KAAK;gBACd,OAAO,EAAE,uBAAuB,KAAK,CAAC,MAAM,EAAE;aAC9C,CAAC;QACH,CAAC;QAED,MAAM,GAAG,GAAG,MAAM,OAAO,CAAC,cAAc,CAAC,KAAK,CAAC,CAAC;QAEhD,IAAI,GAAG,CAAC,IAAI,KAAK,WAAW,EAAE,CAAC;YAC9B,MAAM,EAAE,IAAI,EAAE,CAAC,mCAAmC,EAAE;gBACnD,GAAG,EAAE,GAAG,CAAC,GAAG;gBACZ,KAAK,EAAE,KAAK,CAAC,KAAK;aAClB,CAAC,CAAC;YACH,OAAO;gBACN,OAAO,EAAE,KAAK;gBACd,OAAO,EAAE,0BAA0B;aACnC,CAAC;QACH,CAAC;QAED,IAAI,GAAG,CAAC,IAAI,KAAK,OAAO,EAAE,CAAC;YAC1B,OAAO;gBACN,OAAO,EAAE,KAAK;gBACd,OAAO,EAAE,GAAG,CAAC,OAAO;aACpB,CAAC;QACH,CAAC;QAED,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACvB,MAAM,EAAE,IAAI,EAAE,CAAC,mBAAmB,EAAE,EAAE,GAAG,EAAE,GAAG,CAAC,GAAG,EAAE,KAAK,EAAE,KAAK,CAAC,KAAK,EAAE,CAAC,CAAC;QAE1E,OAAO;YACN,OAAO,EAAE,IAAI;YACb,OAAO,EAAE,gBAAgB;YACzB,GAAG,EAAE,GAAG,CAAC,GAAG;SACZ,CAAC;IACH,CAAC;IAED,KAAK,UAAU,WAAW,CACzB,OAAiB,EACjB,SAAS,GAAG,KAAK;QAEjB,IAAI,CAAC;YACJ,MAAM,EAAE,IAAI,EAAE,CAAC,8BAA8B,EAAE;gBAC9C,MAAM,EAAE,OAAO,CAAC,KAAK;aACrB,CAAC,CAAC;YAEH,MAAM,EAAE,SAAS,EAAE,GAAG,MAAM,aAAa,CACxC;gBACC,MAAM,EAAE,OAAO,CAAC,KAAK;gBACrB,OAAO;gBACP,OAAO,EAAE,OAAO,CAAC,OAAO;gBACxB,SAAS;gBACT,SAAS;aACT,EACD,OAAO,CACP,CAAC;YAEF,OAAO;gBACN,OAAO,EAAE,IAAI;gBACb,OAAO,EAAE,oBAAoB,SAAS,EAAE;gBACxC,SAAS;aACT,CAAC;QACH,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YAChB,MAAM,YAAY,GACjB,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;YACxD,MAAM,EAAE,KAAK,EAAE,CAAC,6BAA6B,YAAY,EAAE,CAAC,CAAC;YAC7D,OAAO;gBACN,OAAO,EAAE,KAAK;gBACd,OAAO,EAAE,YAAY;aACrB,CAAC;QACH,CAAC;IACF,CAAC;IAED,OAAO;QACN,GAAG;QACH,WAAW;KACX,CAAC;AACH,CAAC"}

package/dist/index.d.ts

@@ -1,4 +1,5 @@
 export * from "./types.js";
 export type { Greptor } from "./greptor.js";
 export { createGreptor } from "./greptor.js";
+export type { LanguageModel } from "ai";
 //# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,YAAY,CAAC;AAE3B,YAAY,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,aAAa,EAAE,MAAM,cAAc,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,YAAY,CAAC;AAE3B,YAAY,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,aAAa,EAAE,MAAM,cAAc,CAAC;AAG7C,YAAY,EAAE,aAAa,EAAE,MAAM,IAAI,CAAC"}

package/dist/processing/processor.d.ts

@@ -1,10 +1,10 @@
-import type { LlmClient } from "../llm/llm-factory.js";
+import { type LanguageModel } from "ai";
 import type { DocumentRef, FileStorage } from "../storage/index.js";
 import type { Logger } from "../types.js";
 export interface ProcessorContext {
     domain: string;
-    metadataSchema: string;
-    llm: LlmClient;
+    tagSchema: string;
+    model: LanguageModel;
     storage: FileStorage;
     logger?: Logger;
 }

package/dist/processing/processor.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"processor.d.ts","sourceRoot":"","sources":["../../src/processing/processor.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,uBAAuB,CAAC;AACvD,OAAO,KAAK,EAAE,WAAW,EAAE,WAAW,EAAE,MAAM,qBAAqB,CAAC;AACpE,OAAO,KAAK,EAAE,MAAM,EAAY,MAAM,aAAa,CAAC;AAMpD,MAAM,WAAW,gBAAgB;IAChC,MAAM,EAAE,MAAM,CAAC;IACf,cAAc,EAAE,MAAM,CAAC;IACvB,GAAG,EAAE,SAAS,CAAC;IACf,OAAO,EAAE,WAAW,CAAC;IACrB,MAAM,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,eAAe;IAC/B,OAAO,EAAE,CAAC,GAAG,EAAE,WAAW,KAAK,IAAI,CAAC;IACpC,OAAO,EAAE,MAAM,WAAW,GAAG,SAAS,CAAC;IACvC,IAAI,EAAE,MAAM,MAAM,CAAC;CACnB;AAED,wBAAgB,qBAAqB,IAAI,eAAe,CAgBvD;AAgFD,wBAAgB,sBAAsB,CAAC,IAAI,EAAE;IAC5C,GAAG,EAAE,gBAAgB,CAAC;IACtB,KAAK,EAAE,eAAe,CAAC;IACvB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,WAAW,CAAC,EAAE,MAAM,CAAC;CACrB,GAAG,IAAI,CAkCP;AAED,wBAAsB,2BAA2B,CAAC,IAAI,EAAE;IACvD,OAAO,EAAE,WAAW,CAAC;IACrB,KAAK,EAAE,eAAe,CAAC;IACvB,MAAM,CAAC,EAAE,MAAM,CAAC;CAChB,GAAG,OAAO,CAAC,MAAM,CAAC,CAalB"}
+{"version":3,"file":"processor.d.ts","sourceRoot":"","sources":["../../src/processing/processor.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,aAAa,EAAgB,MAAM,IAAI,CAAC;AAEtD,OAAO,KAAK,EAAE,WAAW,EAAE,WAAW,EAAE,MAAM,qBAAqB,CAAC;AACpE,OAAO,KAAK,EAAE,MAAM,EAAQ,MAAM,aAAa,CAAC;AAIhD,MAAM,WAAW,gBAAgB;IAChC,MAAM,EAAE,MAAM,CAAC;IACf,SAAS,EAAE,MAAM,CAAC;IAClB,KAAK,EAAE,aAAa,CAAC;IACrB,OAAO,EAAE,WAAW,CAAC;IACrB,MAAM,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,eAAe;IAC/B,OAAO,EAAE,CAAC,GAAG,EAAE,WAAW,KAAK,IAAI,CAAC;IACpC,OAAO,EAAE,MAAM,WAAW,GAAG,SAAS,CAAC;IACvC,IAAI,EAAE,MAAM,MAAM,CAAC;CACnB;AAED,wBAAgB,qBAAqB,IAAI,eAAe,CAgBvD;AAmID,wBAAgB,sBAAsB,CAAC,IAAI,EAAE;IAC5C,GAAG,EAAE,gBAAgB,CAAC;IACtB,KAAK,EAAE,eAAe,CAAC;IACvB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,WAAW,CAAC,EAAE,MAAM,CAAC;CACrB,GAAG,IAAI,CAkCP;AAED,wBAAsB,2BAA2B,CAAC,IAAI,EAAE;IACvD,OAAO,EAAE,WAAW,CAAC;IACrB,KAAK,EAAE,eAAe,CAAC;IACvB,MAAM,CAAC,EAAE,MAAM,CAAC;CAChB,GAAG,OAAO,CAAC,MAAM,CAAC,CAalB"}

package/dist/processing/processor.js

@@ -1,6 +1,5 @@
+import { generateText } from "ai";
 import YAML from "yaml";
-import { chunk as chunkDocument } from "./chunk.js";
-import { extractMetadata } from "./extract-metadata.js";
 const DEFAULT_IDLE_SLEEP_MS = 750;
 export function createProcessingQueue() {
     const items = [];
@@ -16,12 +15,8 @@ export function createProcessingQueue() {
         },
     };
 }
-function renderProcessedDocument(metadata, chunkMetadata, chunkContent) {
-    const combinedMetadata = {
-        ...metadata,
-        chunks: chunkMetadata,
-    };
-    const doc = new YAML.Document(combinedMetadata);
+function renderProcessedDocument(tags, chunkContent) {
+    const doc = new YAML.Document(tags);
     YAML.visit(doc, {
         Seq(_, node) {
             const allScalars = node.items.every((item) => YAML.isScalar(item));
@@ -30,33 +25,94 @@ function renderProcessedDocument(metadata, chunkMetadata, chunkContent) {
             }
         },
     });
-    const renderedMetadata = doc.toString({ lineWidth: 200 });
-    return [
-        "---",
-        renderedMetadata.trimEnd(),
-        "---",
-        "",
-        chunkContent.trim(),
-    ].join("\n");
+    const renderedTags = doc.toString({ lineWidth: 200 });
+    return ["---", renderedTags.trimEnd(), "---", "", chunkContent.trim()].join("\n");
+}
+function createProcessingPrompt(rawContent, domain, tagSchema) {
+    return `
+# INSTRUCTIONS
+Clean, chunk, and tag the raw content for **grep-based search** in the domain: ${domain}.
+
+## Core Principle
+Optimize for **single-pass grep scanning**: a single grep hit should reveal what a chunk is about without reading other chunks.
+
+## Objectives
+- Remove noise and boilerplate: ads, sponsors, intros/outros, CTAs, repetitions, contact or social links, and sign-offs.
+- Preserve **all meaning and factual detail exactly** (facts, names, dates, numbers, ranges, uncertainty, conditions, and meaningful URLs).
+- Use **minimal wording** while keeping all information.
+- Chunk the content into **semantic sections** (prefer fewer, richer chunks when possible; do not pad content to reach size targets).
+
+## Output Format (Markdown only)
+
+\`\`\`md
+## 01 Short descriptive title for chunk 1
+field_1=value_1,value_4
+field_2=value_2,
+field_3=value_3,
+<cleaned, condensed content>
+
+## 02 Short descriptive title for chunk 2
+field_1=value_1
+field_4=value_4
+field_5=value_5,value_6
+<cleaned, condensed content>
+\`\`\`
+
+## Tagging Rules
+- Use ONLY fields defined in the SCHEMA (field names must exactly match schema).
+- Do not invent new fields.
+- Omit fields with no value.
+- One tag field per line.
+- DO NOT duplicate fields. For arrays, use comma-separated values.
+- For enums, use only allowed enum values from the schema.
+- Use ISO-8601 for dates (YYYY-MM-DD).
+- Keep tag values grep-friendly:
+	- snake_case where appropriate
+	- tickers, codes, and symbols in UPPERCASE
+- Maintain a tag order as per schema.
+
+## Content Rules
+- Output MUST be plain text or Markdown with simple formatting (headings, lists, bold/italic).
+- Rewrite content to be token-efficient and grep-efficient without altering meaning.
+- Content MUST be split into short paragraphs separated by blank lines.
+- Each paragraph MUST be 1-3 sentences.
+- Each sentence MUST be declarative and information-dense.
+- Keep entities, tickers, and terms explicit; avoid pronouns.
+- Normalize numbers (e.g. "1,000,000.00", "24%").
+- Preserve uncertainty, ranges, and conditional statements exactly.
+- Do not add interpretation, synthesis, or analysis.
+
+# TAG SCHEMA:
+${tagSchema}
+
+# RAW CONTENT:
+${rawContent}
+`;
 }
 async function processDocument(ref, ctx) {
     // 1. Read raw content
-    const { metadata, content } = await ctx.storage.readRawContent(ref);
-    const contentLength = content.length;
-    // 2. Chunk content with LLM
-    ctx.logger?.debug?.("Chunking document", { ref, step: "chunk" });
-    const chunkContent = await chunkDocument(content, ctx.domain, ctx.llm);
-    // 3. Extract metadata with LLM
-    ctx.logger?.debug?.("Extracting metadata", { ref, step: "metadata" });
-    const chunkMetadata = await extractMetadata(chunkContent, ctx.domain, ctx.metadataSchema, ctx.llm);
-    // 4. Parse chunk metadata and render final content
-    const rendered = renderProcessedDocument(metadata, chunkMetadata, chunkContent);
-    // 5. Save processed content
+    const { tags, content } = await ctx.storage.readRawContent(ref);
+    // 2. Clean + chunk + tag with a single LLM call
+    ctx.logger?.debug?.("Processing document", { ref, step: "single-pass" });
+    const prompt = createProcessingPrompt(content, ctx.domain, ctx.tagSchema);
+    const { text, usage } = await generateText({
+        model: ctx.model,
+        prompt,
+    });
+    if (!text) {
+        throw new Error("Failed to process content: empty LLM response");
+    }
+    // 3. Render final content with document-level YAML only
+    const rendered = renderProcessedDocument(tags, text);
+    // 4. Save processed content
     await ctx.storage.saveProcessedContent(ref, rendered);
     ctx.logger?.info?.("Document processed", {
         ref,
-        chunks: chunkMetadata.length,
-        bytes: contentLength,
+        inputCacheReadTokens: usage?.inputTokenDetails.cacheReadTokens,
+        inputCacheWriteTokens: usage?.inputTokenDetails.cacheWriteTokens,
+        inputTokens: usage?.inputTokens,
+        outputTokens: usage?.outputTokens,
+        totalTokens: usage?.totalTokens,
     });
 }
 function sleep(ms) {

package/dist/processing/processor.js.map

@@ -1 +1 @@
-{"version":3,"file":"processor.js","sourceRoot":"","sources":["../../src/processing/processor.ts"],"names":[],"mappings":"AAAA,OAAO,IAAI,MAAM,MAAM,CAAC;AAIxB,OAAO,EAAE,KAAK,IAAI,aAAa,EAAE,MAAM,YAAY,CAAC;AACpD,OAAO,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AAExD,MAAM,qBAAqB,GAAG,GAAG,CAAC;AAgBlC,MAAM,UAAU,qBAAqB;IACpC,MAAM,KAAK,GAAkB,EAAE,CAAC;IAEhC,OAAO;QACN,OAAO,CAAC,GAAG;YACV,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QACjB,CAAC;QAED,IAAI;YACH,OAAO,KAAK,CAAC,MAAM,CAAC;QACrB,CAAC;QAED,OAAO;YACN,OAAO,KAAK,CAAC,KAAK,EAAE,CAAC;QACtB,CAAC;KACD,CAAC;AACH,CAAC;AAED,SAAS,uBAAuB,CAC/B,QAAkB,EAClB,aAAyB,EACzB,YAAoB;IAEpB,MAAM,gBAAgB,GAAG;QACxB,GAAG,QAAQ;QACX,MAAM,EAAE,aAAa;KACrB,CAAC;IAEF,MAAM,GAAG,GAAG,IAAI,IAAI,CAAC,QAAQ,CAAC,gBAAgB,CAAC,CAAC;IAEhD,IAAI,CAAC,KAAK,CAAC,GAAG,EAAE;QACf,GAAG,CAAC,CAAC,EAAE,IAAI;YACV,MAAM,UAAU,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC;YACnE,IAAI,UAAU,EAAE,CAAC;gBAChB,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;YAClB,CAAC;QACF,CAAC;KACD,CAAC,CAAC;IAEH,MAAM,gBAAgB,GAAG,GAAG,CAAC,QAAQ,CAAC,EAAE,SAAS,EAAE,GAAG,EAAE,CAAC,CAAC;IAE1D,OAAO;QACN,KAAK;QACL,gBAAgB,CAAC,OAAO,EAAE;QAC1B,KAAK;QACL,EAAE;QACF,YAAY,CAAC,IAAI,EAAE;KACnB,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACd,CAAC;AAED,KAAK,UAAU,eAAe,CAC7B,GAAgB,EAChB,GAAqB;IAErB,sBAAsB;IACtB,MAAM,EAAE,QAAQ,EAAE,OAAO,EAAE,GAAG,MAAM,GAAG,CAAC,OAAO,CAAC,cAAc,CAAC,GAAG,CAAC,CAAC;IACpE,MAAM,aAAa,GAAG,OAAO,CAAC,MAAM,CAAC;IAErC,4BAA4B;IAC5B,GAAG,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,mBAAmB,EAAE,EAAE,GAAG,EAAE,IAAI,EAAE,OAAO,EAAE,CAAC,CAAC;IACjE,MAAM,YAAY,GAAG,MAAM,aAAa,CAAC,OAAO,EAAE,GAAG,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,CAAC,CAAC;IAEvE,+BAA+B;IAC/B,GAAG,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,qBAAqB,EAAE,EAAE,GAAG,EAAE,IAAI,EAAE,UAAU,EAAE,CAAC,CAAC;IACtE,MAAM,aAAa,GAAG,MAAM,eAAe,CAC1C,YAAY,EACZ,GAAG,CAAC,MAAM,EACV,GAAG,CAAC,cAAc,EAClB,GAAG,CAAC,GAAG,CACP,CAAC;IAEF,mDAAmD;IACnD,MAAM,QAAQ,GAAG,uBAAuB,CACvC,QAAQ,EACR,aAAa,EACb,YAAY,CACZ,CAAC;IAEF,4BAA4B;IAC5B,MAAM,GAAG,CAAC,OAAO,CAAC,oBAAoB,CAAC,GAAG,EAAE,QAAQ,CAAC,CAAC;IAEtD,GAAG,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,oBAAoB,EAAE;QACxC,GAAG;QACH,MAAM,EAAE,aAAa,CAAC,MAAM;QAC5B,KAAK,EAAE,aAAa;KACpB,CAAC,CAAC;AACJ,CAAC;AAED,SAAS,KAAK,CAAC,EAAU;IACxB,OAAO,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,EAAE;QAC9B,MAAM,CAAC,GAAG,UAAU,CAAC,OAAO,EAAE,EAAE,CAAC,CAAC;QAClC,kEAAkE;QACjE,CAAuC,CAAC,KAAK,EAAE,EAAE,CAAC;IACpD,CAAC,CAAC,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,sBAAsB,CAAC,IAKtC;IACA,MAAM,WAAW,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,WAAW,IAAI,CAAC,CAAC,CAAC;IACvD,MAAM,WAAW,GAAG,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,IAAI,CAAC,WAAW,IAAI,qBAAqB,CAAC,CAAC;IAC5E,MAAM,EAAE,GAAG,EAAE,KAAK,EAAE,GAAG,IAAI,CAAC;IAE5B,KAAK,UAAU,UAAU,CAAC,WAAmB;QAC5C,OAAO,IAAI,EAAE,CAAC;YACb,MAAM,MAAM,GAAG,KAAK,CAAC,OAAO,EAAE,CAAC;YAC/B,IAAI,CAAC,MAAM,EAAE,CAAC;gBACb,MAAM,KAAK,CAAC,WAAW,CAAC,CAAC;gBACzB,SAAS;YACV,CAAC;YAED,GAAG,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,oBAAoB,EAAE;gBACzC,MAAM,EAAE,WAAW;gBACnB,GAAG,EAAE,MAAM;aACX,CAAC,CAAC;YACH,IAAI,CAAC;gBACJ,MAAM,eAAe,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;YACpC,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBAChB,GAAG,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,mBAAmB,EAAE;oBACxC,GAAG,EAAE,KAAK;oBACV,GAAG,EAAE,MAAM;oBACX,MAAM,EAAE,WAAW;iBACnB,CAAC,CAAC;YACJ,CAAC;QACF,CAAC;IACF,CAAC;IAED,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,WAAW,EAAE,CAAC,EAAE,EAAE,CAAC;QACtC,UAAU,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IACnB,CAAC;IAED,GAAG,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,4BAA4B,EAAE,EAAE,WAAW,EAAE,CAAC,CAAC;AACpE,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,2BAA2B,CAAC,IAIjD;IACA,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,sBAAsB,EAAE,CAAC;IAEzD,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;QACxB,IAAI,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,6BAA6B,EAAE,EAAE,GAAG,EAAE,CAAC,CAAC;QAC7D,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;IACzB,CAAC;IAED,IAAI,IAAI,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACrB,IAAI,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,6BAA6B,EAAE,EAAE,KAAK,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC,CAAC;IAC7E,CAAC;IAED,OAAO,IAAI,CAAC,MAAM,CAAC;AACpB,CAAC"}
+{"version":3,"file":"processor.js","sourceRoot":"","sources":["../../src/processing/processor.ts"],"names":[],"mappings":"AAAA,OAAO,EAAsB,YAAY,EAAE,MAAM,IAAI,CAAC;AACtD,OAAO,IAAI,MAAM,MAAM,CAAC;AAIxB,MAAM,qBAAqB,GAAG,GAAG,CAAC;AAgBlC,MAAM,UAAU,qBAAqB;IACpC,MAAM,KAAK,GAAkB,EAAE,CAAC;IAEhC,OAAO;QACN,OAAO,CAAC,GAAG;YACV,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QACjB,CAAC;QAED,IAAI;YACH,OAAO,KAAK,CAAC,MAAM,CAAC;QACrB,CAAC;QAED,OAAO;YACN,OAAO,KAAK,CAAC,KAAK,EAAE,CAAC;QACtB,CAAC;KACD,CAAC;AACH,CAAC;AAED,SAAS,uBAAuB,CAAC,IAAU,EAAE,YAAoB;IAChE,MAAM,GAAG,GAAG,IAAI,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC;IAEpC,IAAI,CAAC,KAAK,CAAC,GAAG,EAAE;QACf,GAAG,CAAC,CAAC,EAAE,IAAI;YACV,MAAM,UAAU,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC;YACnE,IAAI,UAAU,EAAE,CAAC;gBAChB,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;YAClB,CAAC;QACF,CAAC;KACD,CAAC,CAAC;IAEH,MAAM,YAAY,GAAG,GAAG,CAAC,QAAQ,CAAC,EAAE,SAAS,EAAE,GAAG,EAAE,CAAC,CAAC;IAEtD,OAAO,CAAC,KAAK,EAAE,YAAY,CAAC,OAAO,EAAE,EAAE,KAAK,EAAE,EAAE,EAAE,YAAY,CAAC,IAAI,EAAE,CAAC,CAAC,IAAI,CAC1E,IAAI,CACJ,CAAC;AACH,CAAC;AAED,SAAS,sBAAsB,CAC9B,UAAkB,EAClB,MAAc,EACd,SAAiB;IAEjB,OAAO;;iFAEyE,MAAM;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;EAoDrF,SAAS;;;EAGT,UAAU;CACX,CAAC;AACF,CAAC;AAED,KAAK,UAAU,eAAe,CAC7B,GAAgB,EAChB,GAAqB;IAErB,sBAAsB;IACtB,MAAM,EAAE,IAAI,EAAE,OAAO,EAAE,GAAG,MAAM,GAAG,CAAC,OAAO,CAAC,cAAc,CAAC,GAAG,CAAC,CAAC;IAEhE,gDAAgD;IAChD,GAAG,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,qBAAqB,EAAE,EAAE,GAAG,EAAE,IAAI,EAAE,aAAa,EAAE,CAAC,CAAC;IACzE,MAAM,MAAM,GAAG,sBAAsB,CAAC,OAAO,EAAE,GAAG,CAAC,MAAM,EAAE,GAAG,CAAC,SAAS,CAAC,CAAC;IAE1E,MAAM,EAAE,IAAI,EAAE,KAAK,EAAE,GAAG,MAAM,YAAY,CAAC;QAC1C,KAAK,EAAE,GAAG,CAAC,KAAK;QAChB,MAAM;KACN,CAAC,CAAC;IAEH,IAAI,CAAC,IAAI,EAAE,CAAC;QACX,MAAM,IAAI,KAAK,CAAC,+CAA+C,CAAC,CAAC;IAClE,CAAC;IAED,wDAAwD;IACxD,MAAM,QAAQ,GAAG,uBAAuB,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IAErD,4BAA4B;IAC5B,MAAM,GAAG,CAAC,OAAO,CAAC,oBAAoB,CAAC,GAAG,EAAE,QAAQ,CAAC,CAAC;IAEtD,GAAG,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,oBAAoB,EAAE;QACxC,GAAG;QACH,oBAAoB,EAAE,KAAK,EAAE,iBAAiB,CAAC,eAAe;QAC9D,qBAAqB,EAAE,KAAK,EAAE,iBAAiB,CAAC,gBAAgB;QAChE,WAAW,EAAE,KAAK,EAAE,WAAW;QAC/B,YAAY,EAAE,KAAK,EAAE,YAAY;QACjC,WAAW,EAAE,KAAK,EAAE,WAAW;KAC/B,CAAC,CAAC;AACJ,CAAC;AAED,SAAS,KAAK,CAAC,EAAU;IACxB,OAAO,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,EAAE;QAC9B,MAAM,CAAC,GAAG,UAAU,CAAC,OAAO,EAAE,EAAE,CAAC,CAAC;QAClC,kEAAkE;QACjE,CAAuC,CAAC,KAAK,EAAE,EAAE,CAAC;IACpD,CAAC,CAAC,CAAC;AACJ,CAAC;AAED,MAAM,UAAU,sBAAsB,CAAC,IAKtC;IACA,MAAM,WAAW,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,WAAW,IAAI,CAAC,CAAC,CAAC;IACvD,MAAM,WAAW,GAAG,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,IAAI,CAAC,WAAW,IAAI,qBAAqB,CAAC,CAAC;IAC5E,MAAM,EAAE,GAAG,EAAE,KAAK,EAAE,GAAG,IAAI,CAAC;IAE5B,KAAK,UAAU,UAAU,CAAC,WAAmB;QAC5C,OAAO,IAAI,EAAE,CAAC;YACb,MAAM,MAAM,GAAG,KAAK,CAAC,OAAO,EAAE,CAAC;YAC/B,IAAI,CAAC,MAAM,EAAE,CAAC;gBACb,MAAM,KAAK,CAAC,WAAW,CAAC,CAAC;gBACzB,SAAS;YACV,CAAC;YAED,GAAG,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,oBAAoB,EAAE;gBACzC,MAAM,EAAE,WAAW;gBACnB,GAAG,EAAE,MAAM;aACX,CAAC,CAAC;YACH,IAAI,CAAC;gBACJ,MAAM,eAAe,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;YACpC,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBAChB,GAAG,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,mBAAmB,EAAE;oBACxC,GAAG,EAAE,KAAK;oBACV,GAAG,EAAE,MAAM;oBACX,MAAM,EAAE,WAAW;iBACnB,CAAC,CAAC;YACJ,CAAC;QACF,CAAC;IACF,CAAC;IAED,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,WAAW,EAAE,CAAC,EAAE,EAAE,CAAC;QACtC,UAAU,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IACnB,CAAC;IAED,GAAG,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,4BAA4B,EAAE,EAAE,WAAW,EAAE,CAAC,CAAC;AACpE,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,2BAA2B,CAAC,IAIjD;IACA,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,sBAAsB,EAAE,CAAC;IAEzD,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;QACxB,IAAI,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,6BAA6B,EAAE,EAAE,GAAG,EAAE,CAAC,CAAC;QAC7D,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;IACzB,CAAC;IAED,IAAI,IAAI,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACrB,IAAI,CAAC,MAAM,EAAE,KAAK,EAAE,CAAC,6BAA6B,EAAE,EAAE,KAAK,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC,CAAC;IAC7E,CAAC;IAED,OAAO,IAAI,CAAC,MAAM,CAAC;AACpB,CAAC"}

package/dist/skills/skill-generator.d.ts

@@ -1,10 +1,10 @@
 import type { FileStorage } from "../storage/file-storage.js";
-import type { MetadataSchemaItem } from "../types.js";
+import type { TagSchemaItem } from "../types.js";
 export interface SkillGeneratorOptions {
     domain: string;
     sources: string[];
     baseDir: string;
-    metadataSchema: MetadataSchemaItem[];
+    tagSchema: TagSchemaItem[];
     overwrite: boolean;
 }
 /**

package/dist/skills/skill-generator.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"skill-generator.d.ts","sourceRoot":"","sources":["../../src/skills/skill-generator.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,4BAA4B,CAAC;AAC9D,OAAO,KAAK,EAAE,kBAAkB,EAAE,MAAM,aAAa,CAAC;AAGtD,MAAM,WAAW,qBAAqB;IACrC,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;IAChB,cAAc,EAAE,kBAAkB,EAAE,CAAC;IACrC,SAAS,EAAE,OAAO,CAAC;CACnB;AAqND;;GAEG;AACH,wBAAsB,aAAa,CAClC,OAAO,EAAE,qBAAqB,EAC9B,WAAW,EAAE,WAAW,GACtB,OAAO,CAAC;IAAE,SAAS,EAAE,MAAM,CAAA;CAAE,CAAC,CAsBhC"}
+{"version":3,"file":"skill-generator.d.ts","sourceRoot":"","sources":["../../src/skills/skill-generator.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,4BAA4B,CAAC;AAC9D,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAGjD,MAAM,WAAW,qBAAqB;IACrC,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;IAChB,SAAS,EAAE,aAAa,EAAE,CAAC;IAC3B,SAAS,EAAE,OAAO,CAAC;CACnB;AA2WD;;GAEG;AACH,wBAAsB,aAAa,CAClC,OAAO,EAAE,qBAAqB,EAC9B,WAAW,EAAE,WAAW,GACtB,OAAO,CAAC;IAAE,SAAS,EAAE,MAAM,CAAA;CAAE,CAAC,CAsBhC"}