npm - @robthepcguy/rag-vault - Versions diffs - 1.0.0 - Mend

@robthepcguy/rag-vault 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (131) hide show

package/LICENSE +24 -0
package/README.md +421 -0
package/dist/bin/install-skills.d.ts +20 -0
package/dist/bin/install-skills.d.ts.map +1 -0
package/dist/bin/install-skills.js +196 -0
package/dist/bin/install-skills.js.map +1 -0
package/dist/chunker/index.d.ts +11 -0
package/dist/chunker/index.d.ts.map +1 -0
package/dist/chunker/index.js +6 -0
package/dist/chunker/index.js.map +1 -0
package/dist/chunker/semantic-chunker.d.ts +96 -0
package/dist/chunker/semantic-chunker.d.ts.map +1 -0
package/dist/chunker/semantic-chunker.js +267 -0
package/dist/chunker/semantic-chunker.js.map +1 -0
package/dist/chunker/sentence-splitter.d.ts +16 -0
package/dist/chunker/sentence-splitter.d.ts.map +1 -0
package/dist/chunker/sentence-splitter.js +114 -0
package/dist/chunker/sentence-splitter.js.map +1 -0
package/dist/embedder/index.d.ts +55 -0
package/dist/embedder/index.d.ts.map +1 -0
package/dist/embedder/index.js +146 -0
package/dist/embedder/index.js.map +1 -0
package/dist/errors/index.d.ts +73 -0
package/dist/errors/index.d.ts.map +1 -0
package/dist/errors/index.js +170 -0
package/dist/errors/index.js.map +1 -0
package/dist/index.d.ts +3 -0
package/dist/index.d.ts.map +1 -0
package/dist/index.js +91 -0
package/dist/index.js.map +1 -0
package/dist/parser/html-parser.d.ts +14 -0
package/dist/parser/html-parser.d.ts.map +1 -0
package/dist/parser/html-parser.js +99 -0
package/dist/parser/html-parser.js.map +1 -0
package/dist/parser/index.d.ts +144 -0
package/dist/parser/index.d.ts.map +1 -0
package/dist/parser/index.js +446 -0
package/dist/parser/index.js.map +1 -0
package/dist/parser/pdf-filter.d.ts +89 -0
package/dist/parser/pdf-filter.d.ts.map +1 -0
package/dist/parser/pdf-filter.js +304 -0
package/dist/parser/pdf-filter.js.map +1 -0
package/dist/server/index.d.ts +144 -0
package/dist/server/index.d.ts.map +1 -0
package/dist/server/index.js +518 -0
package/dist/server/index.js.map +1 -0
package/dist/server/raw-data-utils.d.ts +81 -0
package/dist/server/raw-data-utils.d.ts.map +1 -0
package/dist/server/raw-data-utils.js +196 -0
package/dist/server/raw-data-utils.js.map +1 -0
package/dist/server/schemas.d.ts +186 -0
package/dist/server/schemas.d.ts.map +1 -0
package/dist/server/schemas.js +99 -0
package/dist/server/schemas.js.map +1 -0
package/dist/utils/config-parsers.d.ts +14 -0
package/dist/utils/config-parsers.d.ts.map +1 -0
package/dist/utils/config-parsers.js +47 -0
package/dist/utils/config-parsers.js.map +1 -0
package/dist/utils/config.d.ts +37 -0
package/dist/utils/config.d.ts.map +1 -0
package/dist/utils/config.js +52 -0
package/dist/utils/config.js.map +1 -0
package/dist/utils/logger.d.ts +36 -0
package/dist/utils/logger.d.ts.map +1 -0
package/dist/utils/logger.js +64 -0
package/dist/utils/logger.js.map +1 -0
package/dist/utils/math.d.ts +34 -0
package/dist/utils/math.d.ts.map +1 -0
package/dist/utils/math.js +73 -0
package/dist/utils/math.js.map +1 -0
package/dist/utils/process-handlers.d.ts +26 -0
package/dist/utils/process-handlers.d.ts.map +1 -0
package/dist/utils/process-handlers.js +69 -0
package/dist/utils/process-handlers.js.map +1 -0
package/dist/vectordb/index.d.ts +210 -0
package/dist/vectordb/index.d.ts.map +1 -0
package/dist/vectordb/index.js +613 -0
package/dist/vectordb/index.js.map +1 -0
package/dist/web/api-routes.d.ts +9 -0
package/dist/web/api-routes.d.ts.map +1 -0
package/dist/web/api-routes.js +127 -0
package/dist/web/api-routes.js.map +1 -0
package/dist/web/config-routes.d.ts +7 -0
package/dist/web/config-routes.d.ts.map +1 -0
package/dist/web/config-routes.js +54 -0
package/dist/web/config-routes.js.map +1 -0
package/dist/web/database-manager.d.ts +130 -0
package/dist/web/database-manager.d.ts.map +1 -0
package/dist/web/database-manager.js +382 -0
package/dist/web/database-manager.js.map +1 -0
package/dist/web/http-server.d.ts +28 -0
package/dist/web/http-server.d.ts.map +1 -0
package/dist/web/http-server.js +311 -0
package/dist/web/http-server.js.map +1 -0
package/dist/web/index.d.ts +3 -0
package/dist/web/index.d.ts.map +1 -0
package/dist/web/index.js +114 -0
package/dist/web/index.js.map +1 -0
package/dist/web/middleware/async-handler.d.ts +17 -0
package/dist/web/middleware/async-handler.d.ts.map +1 -0
package/dist/web/middleware/async-handler.js +26 -0
package/dist/web/middleware/async-handler.js.map +1 -0
package/dist/web/middleware/auth.d.ts +22 -0
package/dist/web/middleware/auth.d.ts.map +1 -0
package/dist/web/middleware/auth.js +81 -0
package/dist/web/middleware/auth.js.map +1 -0
package/dist/web/middleware/error-handler.d.ts +36 -0
package/dist/web/middleware/error-handler.d.ts.map +1 -0
package/dist/web/middleware/error-handler.js +68 -0
package/dist/web/middleware/error-handler.js.map +1 -0
package/dist/web/middleware/index.d.ts +6 -0
package/dist/web/middleware/index.d.ts.map +1 -0
package/dist/web/middleware/index.js +19 -0
package/dist/web/middleware/index.js.map +1 -0
package/dist/web/middleware/rate-limit.d.ts +38 -0
package/dist/web/middleware/rate-limit.d.ts.map +1 -0
package/dist/web/middleware/rate-limit.js +116 -0
package/dist/web/middleware/rate-limit.js.map +1 -0
package/dist/web/middleware/request-logger.d.ts +52 -0
package/dist/web/middleware/request-logger.d.ts.map +1 -0
package/dist/web/middleware/request-logger.js +74 -0
package/dist/web/middleware/request-logger.js.map +1 -0
package/dist/web/types.d.ts +6 -0
package/dist/web/types.d.ts.map +1 -0
package/dist/web/types.js +4 -0
package/dist/web/types.js.map +1 -0
package/package.json +135 -0
package/skills/rag-vault/SKILL.md +111 -0
package/skills/rag-vault/references/html-ingestion.md +73 -0
package/skills/rag-vault/references/query-optimization.md +57 -0
package/skills/rag-vault/references/result-refinement.md +54 -0

package/skills/rag-vault/SKILL.md ADDED Viewed

@@ -0,0 +1,111 @@
+---
+name: rag-vault
+description: This skill should be used when the user asks to "search documents", "query RAG", "ingest file", "ingest PDF", "save web page", "add to knowledge base", or mentions document search, semantic search, vector search, or RAG operations. Provides score interpretation (< 0.3 good, > 0.5 skip), query optimization, and ingestion guidance for query_documents, ingest_file, ingest_data tools.
+version: 1.0.0
+---
+# RAG Vault Skills
+## Tools
+| Tool | Use When |
+|------|----------|
+| `ingest_file` | Local files (PDF, DOCX, TXT, MD, JSON, JSONL) |
+| `ingest_data` | Raw content (HTML, text) with source URL |
+| `query_documents` | Semantic + keyword hybrid search |
+| `delete_file` / `list_files` / `status` | Management |
+## Search: Core Rules
+Hybrid search combines vector (semantic) and keyword (BM25).
+### Score Interpretation
+Lower = better match. Use this to filter noise.
+| Score | Action |
+|-------|--------|
+| < 0.3 | Use directly |
+| 0.3-0.5 | Include if mentions same concept/entity |
+| > 0.5 | Skip unless no better results |
+### Limit Selection
+| Intent | Limit |
+|--------|-------|
+| Specific answer (function, error) | 5 |
+| General understanding | 10 |
+| Comprehensive survey | 20 |
+### Query Formulation
+| Situation | Why Transform | Action |
+|-----------|---------------|--------|
+| Specific term mentioned | Keyword search needs exact match | KEEP term |
+| Vague query | Vector search needs semantic signal | ADD context |
+| Error stack or code block | Long text dilutes relevance | EXTRACT core keywords |
+| Multiple distinct topics | Single query conflates results | SPLIT queries |
+| Few/poor results | Term mismatch | EXPAND (see below) |
+### Query Expansion
+When results are few or all score > 0.5, expand query terms:
+- Keep original term first, add 2-4 variants
+- Types: synonyms, abbreviations, related terms, word forms
+- Example: `"config"` → `"config configuration settings configure"`
+Avoid over-expansion (causes topic drift).
+### Result Selection
+When to include vs skip—based on answer quality, not just score.
+**INCLUDE** if:
+- Directly answers the question
+- Provides necessary context
+- Score < 0.5
+**SKIP** if:
+- Same keyword, unrelated context
+- Score > 0.7
+- Mentions term without explanation
+## Ingestion
+### ingest_file
+```
+ingest_file({ filePath: "/absolute/path/to/document.pdf" })
+```
+### ingest_data
+```
+ingest_data({
+  content: "<html>...</html>",
+  metadata: { source: "https://example.com/page", format: "html" }
+})
+```
+**Format selection** — match the data you have:
+- HTML string → `format: "html"`
+- Markdown string → `format: "markdown"`
+- Other → `format: "text"`
+**Source format:**
+- Web page → Use URL: `https://example.com/page`
+- Other content → Use scheme: `{type}://{date}` or `{type}://{date}/{detail}`
+  - Examples: `clipboard://2024-12-30`, `chat://2024-12-30/project-discussion`
+**HTML source options:**
+- Static page → LLM fetch
+- SPA/JS-rendered → Browser MCP
+- Auth required → Manual paste
+Re-ingest same source to update. Use same source in `delete_file` to remove.
+## References
+For edge cases and examples:
+- [html-ingestion.md](references/html-ingestion.md) - URL normalization, SPA handling
+- [query-optimization.md](references/query-optimization.md) - Query patterns by intent
+- [result-refinement.md](references/result-refinement.md) - Contradiction resolution, chunking

package/skills/rag-vault/references/html-ingestion.md ADDED Viewed

@@ -0,0 +1,73 @@
+# HTML Ingestion Reference
+Basic usage is in SKILL.md. This covers URL handling and edge cases.
+## System Behavior
+The parser extracts main content only—navigation, ads, and boilerplate are stripped. What gets indexed is clean body text, not the full HTML.
+## When to Use Each Source Method
+| Source Type | Method | Why |
+|-------------|--------|-----|
+| Static page, public | LLM fetch | Simplest, no extra tools |
+| SPA / JS-rendered | Browser MCP | Need rendered DOM |
+| Auth required | Manual paste | Can't fetch programmatically |
+## URL Normalization
+System strips query strings and fragments:
+```
+https://example.com/page?utm=x#section → https://example.com/page
+```
+**When query strings matter** (pagination, dynamic IDs):
+```
+ingest_data({
+  content: page1_html,
+  metadata: { source: "https://example.com/results?page=1", format: "html" }
+})
+```
+Explicitly include full URL as source.
+## Edge Cases
+### Empty/Minimal Extraction
+Why it happens:
+- JS-rendered content (use browser MCP)
+- Non-standard HTML structure
+- Login required
+### SPA/Dynamic Content
+1. Use browser MCP to render
+2. Wait for content load
+3. Extract rendered HTML
+4. Ingest via `ingest_data`
+### Pages with Only Navigation
+Skip or fetch deeper linked pages instead.
+## Updating Content
+Re-ingest with same source to replace:
+```
+ingest_data({
+  content: updated_html,
+  metadata: { source: "https://example.com/page", format: "html" }
+})
+```
+## Search Results
+Results from HTML include `source` field:
+```json
+{
+  "filePath": "raw-data/abc123.md",
+  "source": "https://example.com/page",
+  "text": "...",
+  "score": 0.25
+}
+```

package/skills/rag-vault/references/query-optimization.md ADDED Viewed

@@ -0,0 +1,57 @@
+# Query Optimization Reference
+Core rules are in SKILL.md. This covers patterns and edge cases.
+## Query Patterns by Intent
+| User Intent | Query Pattern | Why |
+|-------------|---------------|-----|
+| Definition/Concept | `"[term] definition concept"` | Targets explanatory content |
+| How-To/Procedure | `"[action] steps example usage"` | Targets instructional content |
+| API/Function | `"[function] API arguments return"` | Targets reference docs |
+| Troubleshooting | `"[error] fix solution cause"` | Targets problem-solving content |
+## Multi-Query: When to Split
+**Split** when "and" connects distinct topics:
+```
+"How do I authenticate AND handle errors?"
+→ Query 1: "authentication login JWT session"
+→ Query 2: "error handling exception catch"
+```
+**Don't split** when "and" is within single topic:
+```
+"How do I set up and configure the database?"
+→ Single: "database setup configuration"
+```
+## Query Expansion Examples
+When results are few or all score > 0.5:
+| Type | Original | Expanded |
+|------|----------|----------|
+| Synonyms | delete | "delete remove" |
+| Abbreviations | API | "API Application Programming Interface" |
+| Related terms | auth | "auth authentication login" |
+| Word forms | config | "config configuration configure" |
+Keep original term first. Limit to 2-4 additions.
+## Iterative Refinement
+When initial results are unsatisfactory:
+| Problem | Why It Happens | Action |
+|---------|----------------|--------|
+| Too few results | Term mismatch | Expand query (see above) |
+| Too many irrelevant | Query too broad | Add specific terms |
+| Missing expected | Phrasing mismatch | Try alternative wording |
+## Language Mixing
+Ngram tokenization supports cross-language queries:
+```
+"API error handling" → matches both English and Japanese content
+```

package/skills/rag-vault/references/result-refinement.md ADDED Viewed

@@ -0,0 +1,54 @@
+# Result Refinement Reference
+Core rules (score, include/skip) are in SKILL.md. This covers when and how to combine multiple results.
+## When to Synthesize vs Filter
+Match approach to user intent:
+| User Intent | Approach | Why |
+|-------------|----------|-----|
+| Specific answer ("how to X") | Filter to 1-2 best | Extra results add noise |
+| Understanding a topic | Synthesize multiple | Builds complete picture |
+| Troubleshooting error | Filter to direct cause | Tangential info confuses |
+| Comparing options | Synthesize with structure | Need all perspectives |
+## Multiple Results Handling
+### Synthesis
+When: User needs comprehensive understanding.
+```
+Result 1: "API accepts JSON..."
+Result 2: "Auth uses Bearer tokens..."
+→ Combine into unified answer
+```
+### Deduplication
+When: Results overlap significantly.
+1. Pick most complete result
+2. Add only unique info from others
+### Contradiction Resolution
+When: Results conflict.
+Priority: Lower score (= better match)
+If unresolved → Note discrepancy to user
+## Chunk Context
+Single chunks may lack context ("as described above").
+- Note when information is partial
+- Group multiple chunks from same `filePath` as coherent sections
+## No Results
+1. Rephrase query (alternative terms)
+2. Broaden scope
+3. Check ingestion (`list_files`)
+4. Inform user: no matching content