@devilsdev/rag-pipeline-utils 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.dockerignore +14 -0
- package/.github/workflows/ci.yml +42 -0
- package/.github/workflows/release.yml +21 -0
- package/Dockerfile +22 -0
- package/LICENSE +674 -0
- package/README.md +116 -0
- package/__tests__/fixtures/sample-eval-dataset.json +15 -0
- package/__tests__/integration/cli/config-flow.test.js +44 -0
- package/__tests__/integration/config/load-config.test.js +52 -0
- package/__tests__/unit/core/plugin-registry.test.js +55 -0
- package/__tests__/unit/reranker/__snapshots__/reranker.snapshot.test.js.snap +15 -0
- package/__tests__/unit/reranker/llm-reranker.test.js +42 -0
- package/__tests__/unit/reranker/reranker.snapshot.test.js +17 -0
- package/bin/cli.js +168 -0
- package/docs-site/README.md +41 -0
- package/docs-site/blog/2019-05-28-first-blog-post.md +12 -0
- package/docs-site/blog/2019-05-29-long-blog-post.md +44 -0
- package/docs-site/blog/2021-08-01-mdx-blog-post.mdx +24 -0
- package/docs-site/blog/2021-08-26-welcome/docusaurus-plushie-banner.jpeg +0 -0
- package/docs-site/blog/2021-08-26-welcome/index.md +29 -0
- package/docs-site/blog/authors.yml +23 -0
- package/docs-site/blog/tags.yml +19 -0
- package/docs-site/docs/Architecture.md +76 -0
- package/docs-site/docs/CLI.md +69 -0
- package/docs-site/docs/Evaluation.md +61 -0
- package/docs-site/docs/FAQ.md +50 -0
- package/docs-site/docs/Introduction.md +58 -0
- package/docs-site/docs/Plugins.md +61 -0
- package/docs-site/docs/Usage.md +72 -0
- package/docs-site/docusaurus.config.js +57 -0
- package/docs-site/package-lock.json +17914 -0
- package/docs-site/package.json +44 -0
- package/docs-site/sidebars.js +22 -0
- package/docs-site/src/components/HomepageFeatures/index.js +59 -0
- package/docs-site/src/components/HomepageFeatures/styles.module.css +58 -0
- package/docs-site/src/css/custom.css +81 -0
- package/docs-site/src/pages/index.js +37 -0
- package/docs-site/src/pages/index.module.css +23 -0
- package/docs-site/src/pages/markdown-page.md +7 -0
- package/docs-site/static/.nojekyll +0 -0
- package/docs-site/static/img/favicon.ico +0 -0
- package/docs-site/static/img/logo.png +0 -0
- package/docs-site/static/img/logo.svg +1539 -0
- package/docs-site/static/img/social-card.jpg +0 -0
- package/jest.config.js +14 -0
- package/package.json +65 -0
- package/public/index.html +0 -0
- package/public/results/sample-eval.json +0 -0
- package/public/src/App.jsx +57 -0
- package/release.config.js +23 -0
- package/server.js +31 -0
- package/src/config/load-config.js +35 -0
- package/src/config/validate-schema.js +49 -0
- package/src/core/create-pipeline.js +82 -0
- package/src/core/plugin-registry.js +61 -0
- package/src/dag/dag-engine.js +70 -0
- package/src/evaluate/evaluator.js +44 -0
- package/src/evaluate/scoring.js +74 -0
- package/src/loader/csv-loader.js +30 -0
- package/src/loader/directory-loader.js +38 -0
- package/src/loader/html-loader.js +57 -0
- package/src/loader/markdown-loader.js +58 -0
- package/src/reranker/llm-reranker.js +49 -0
- package/src/utils/logger.js +29 -0
- package/src/utils/retry.js +42 -0
package/README.md
ADDED
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
# @DevilsDev/rag-pipeline-utils
|
|
2
|
+
|
|
3
|
+
[](https://github.com/DevilsDev/rag-pipeline-utils/actions)
|
|
4
|
+
[](https://codecov.io/gh/DevilsDev/rag-pipeline-utils)
|
|
5
|
+
|
|
6
|
+
**A modular Node.js toolkit for building RAG (Retrieval-Augmented Generation) pipelines with clean abstractions, pluggable components, and scalable architecture.**
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## Features
|
|
11
|
+
|
|
12
|
+
- ✅ Plugin-based architecture for RAG pipelines
|
|
13
|
+
- ✅ Embedding + retrieval abstraction
|
|
14
|
+
- ✅ Vector DB integrations (e.g., Pinecone)
|
|
15
|
+
- ✅ Modular document loaders (PDF, HTML, Markdown, CSV, Directory)
|
|
16
|
+
- ✅ CLI with `.ragrc.json` config fallback
|
|
17
|
+
- ✅ Retry, backoff, and logging built-in
|
|
18
|
+
- ✅ Reranker support and evaluation tooling
|
|
19
|
+
- ✅ React dashboard for evaluation results
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## Installation
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
npm install @yourorg/rag-pipeline-utils
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
## Usage
|
|
32
|
+
|
|
33
|
+
```js
|
|
34
|
+
import { createRagPipeline } from '@yourorg/rag-pipeline-utils';
|
|
35
|
+
|
|
36
|
+
const rag = createRagPipeline({
|
|
37
|
+
loader: 'markdown',
|
|
38
|
+
embedder: 'openai',
|
|
39
|
+
retriever: 'pinecone',
|
|
40
|
+
llm: 'openai-gpt-4',
|
|
41
|
+
useReranker: true
|
|
42
|
+
});
|
|
43
|
+
|
|
44
|
+
const answer = await rag.query('What is a vector database?');
|
|
45
|
+
console.log(answer);
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
---
|
|
49
|
+
|
|
50
|
+
## Testing
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
npm test
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
---
|
|
57
|
+
|
|
58
|
+
## CLI
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
rag-pipeline ingest ./docs/my.pdf --loader pdf
|
|
62
|
+
rag-pipeline query "Explain RAG pipelines" --llm openai-gpt-4
|
|
63
|
+
rag-pipeline evaluate ./fixtures/sample-eval-dataset.json
|
|
64
|
+
rag-pipeline rerank "Explain embeddings" --retriever pinecone
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Supports `.ragrc.json` fallback configuration:
|
|
68
|
+
|
|
69
|
+
```json
|
|
70
|
+
{
|
|
71
|
+
"loader": "directory",
|
|
72
|
+
"embedder": "openai",
|
|
73
|
+
"retriever": "pinecone",
|
|
74
|
+
"llm": "openai-gpt-4",
|
|
75
|
+
"useReranker": true
|
|
76
|
+
}
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
|
|
81
|
+
## Project Structure
|
|
82
|
+
|
|
83
|
+
```
|
|
84
|
+
/src
|
|
85
|
+
├── core/ # Pipeline + registry
|
|
86
|
+
├── loader/ # Loaders for md, html, csv, dir
|
|
87
|
+
├── reranker/ # LLM-based context reranker
|
|
88
|
+
├── evaluate/ # Scoring, batch QA evaluation
|
|
89
|
+
├── utils/ # Logger, retry, schema
|
|
90
|
+
/bin/cli.js # CLI entrypoint
|
|
91
|
+
/__tests__/ # Unit, integration, snapshot tests
|
|
92
|
+
/public/ # Evaluation dashboard (React)
|
|
93
|
+
.github/workflows/ci.yml # CI config
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
---
|
|
97
|
+
|
|
98
|
+
## Evaluation Dashboard
|
|
99
|
+
|
|
100
|
+
```bash
|
|
101
|
+
node server.js
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Visit: [http://localhost:3000](http://localhost:3000) → Visualizes metrics, BLEU/ROUGE, pass rate.
|
|
105
|
+
|
|
106
|
+
---
|
|
107
|
+
|
|
108
|
+
## License
|
|
109
|
+
|
|
110
|
+
Apache-2.0
|
|
111
|
+
|
|
112
|
+
---
|
|
113
|
+
|
|
114
|
+
## 👤 Author
|
|
115
|
+
|
|
116
|
+
Ali Kahwaji — [@DevilsDev](https://github.com/DevilsDev)
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
[
|
|
2
|
+
{
|
|
3
|
+
"prompt": "What is a vector database?",
|
|
4
|
+
"expected": "A vector database is a system optimized for storing and retrieving data represented as vectors, often used in similarity search."
|
|
5
|
+
},
|
|
6
|
+
{
|
|
7
|
+
"prompt": "Explain what chunking means in RAG.",
|
|
8
|
+
"expected": "Chunking is the process of splitting documents into smaller, manageable pieces for embedding and retrieval in a RAG pipeline."
|
|
9
|
+
},
|
|
10
|
+
{
|
|
11
|
+
"prompt": "What is the role of embedding in a RAG pipeline?",
|
|
12
|
+
"expected": "Embeddings convert text into numerical vector form so they can be compared in a vector database during retrieval."
|
|
13
|
+
}
|
|
14
|
+
]
|
|
15
|
+
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Version: 0.1.0
|
|
3
|
+
* Path: /__tests__/integration/cli/config-flow.test.js
|
|
4
|
+
* Description: Integration tests for CLI fallback to .ragrc.json and full flow execution
|
|
5
|
+
* Author: Ali Kahwaji
|
|
6
|
+
*/
|
|
7
|
+
|
|
8
|
+
import fs from 'fs';
|
|
9
|
+
import path from 'path';
|
|
10
|
+
import { execSync } from 'child_process';
|
|
11
|
+
|
|
12
|
+
const CONFIG_PATH = path.resolve(process.cwd(), '.ragrc.json');
|
|
13
|
+
const CLI_PATH = path.resolve('./bin/cli.js');
|
|
14
|
+
|
|
15
|
+
const validConfig = {
|
|
16
|
+
loader: 'pdf',
|
|
17
|
+
embedder: 'openai',
|
|
18
|
+
retriever: 'pinecone',
|
|
19
|
+
llm: 'openai-gpt-4',
|
|
20
|
+
chunk: {
|
|
21
|
+
strategy: 'semantic',
|
|
22
|
+
size: 256
|
|
23
|
+
}
|
|
24
|
+
};
|
|
25
|
+
|
|
26
|
+
describe('CLI integration with .ragrc.json config fallback', () => {
|
|
27
|
+
beforeEach(() => {
|
|
28
|
+
fs.writeFileSync(CONFIG_PATH, JSON.stringify(validConfig, null, 2));
|
|
29
|
+
});
|
|
30
|
+
|
|
31
|
+
afterEach(() => {
|
|
32
|
+
if (fs.existsSync(CONFIG_PATH)) fs.unlinkSync(CONFIG_PATH);
|
|
33
|
+
});
|
|
34
|
+
|
|
35
|
+
test('executes CLI ingest using config fallback', () => {
|
|
36
|
+
const result = execSync(`node ${CLI_PATH} ingest ./sample.pdf`, { encoding: 'utf-8' });
|
|
37
|
+
expect(result).toContain('Ingestion complete');
|
|
38
|
+
});
|
|
39
|
+
|
|
40
|
+
test('executes CLI query using config fallback', () => {
|
|
41
|
+
const output = execSync(`node ${CLI_PATH} query "What is a vector?"`, { encoding: 'utf-8' });
|
|
42
|
+
expect(output).toContain('Answer:');
|
|
43
|
+
});
|
|
44
|
+
});
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Version: 0.1.0
|
|
3
|
+
* Path: /__tests__/integration/config/load-config.test.js
|
|
4
|
+
* Description: Integration tests for loading and validating .ragrc.json config
|
|
5
|
+
* Author: Ali Kahwaji
|
|
6
|
+
*/
|
|
7
|
+
|
|
8
|
+
import fs from 'fs';
|
|
9
|
+
import path from 'path';
|
|
10
|
+
import { loadRagConfig } from '../../../src/config/load-config.js';
|
|
11
|
+
|
|
12
|
+
const FIXTURE_PATH = path.resolve(process.cwd(), '.ragrc.json');
|
|
13
|
+
|
|
14
|
+
const validConfig = {
|
|
15
|
+
loader: 'pdf',
|
|
16
|
+
embedder: 'openai',
|
|
17
|
+
retriever: 'pinecone',
|
|
18
|
+
llm: 'openai-gpt-4',
|
|
19
|
+
chunk: {
|
|
20
|
+
strategy: 'semantic',
|
|
21
|
+
size: 500
|
|
22
|
+
}
|
|
23
|
+
};
|
|
24
|
+
|
|
25
|
+
const invalidConfig = {
|
|
26
|
+
loader: 'pdf',
|
|
27
|
+
chunk: { size: 'large' } // Missing required keys + invalid type
|
|
28
|
+
};
|
|
29
|
+
|
|
30
|
+
describe('loadRagConfig()', () => {
|
|
31
|
+
afterEach(() => {
|
|
32
|
+
if (fs.existsSync(FIXTURE_PATH)) {
|
|
33
|
+
fs.unlinkSync(FIXTURE_PATH);
|
|
34
|
+
}
|
|
35
|
+
});
|
|
36
|
+
|
|
37
|
+
test('loads a valid .ragrc.json config file successfully', () => {
|
|
38
|
+
fs.writeFileSync(FIXTURE_PATH, JSON.stringify(validConfig, null, 2));
|
|
39
|
+
const result = loadRagConfig();
|
|
40
|
+
expect(result).toMatchObject(validConfig);
|
|
41
|
+
});
|
|
42
|
+
|
|
43
|
+
test('throws validation error for invalid .ragrc.json config', () => {
|
|
44
|
+
fs.writeFileSync(FIXTURE_PATH, JSON.stringify(invalidConfig, null, 2));
|
|
45
|
+
expect(() => loadRagConfig()).toThrow(/Invalid config/);
|
|
46
|
+
});
|
|
47
|
+
|
|
48
|
+
test('throws error if config file does not exist', () => {
|
|
49
|
+
expect(() => loadRagConfig()).toThrow(/Config file not found/);
|
|
50
|
+
});
|
|
51
|
+
});
|
|
52
|
+
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Version: 0.1.0
|
|
3
|
+
* Path: /__tests__/unit/core/plugin-registry.test.js
|
|
4
|
+
* Description: Unit tests for the PluginRegistry class
|
|
5
|
+
* Author: Ali Kahwaji
|
|
6
|
+
*/
|
|
7
|
+
|
|
8
|
+
import { PluginRegistry } from '../../../src/core/plugin-registry.js';
|
|
9
|
+
|
|
10
|
+
describe('PluginRegistry', () => {
|
|
11
|
+
let registry;
|
|
12
|
+
|
|
13
|
+
beforeEach(() => {
|
|
14
|
+
registry = new PluginRegistry();
|
|
15
|
+
});
|
|
16
|
+
|
|
17
|
+
test('registers and retrieves a plugin successfully', () => {
|
|
18
|
+
const mockPlugin = { name: 'test-loader' };
|
|
19
|
+
registry.register('loader', 'test', mockPlugin);
|
|
20
|
+
const result = registry.get('loader', 'test');
|
|
21
|
+
expect(result).toBe(mockPlugin);
|
|
22
|
+
});
|
|
23
|
+
|
|
24
|
+
test('overwrites existing plugin under the same name', () => {
|
|
25
|
+
const firstPlugin = { name: 'first' };
|
|
26
|
+
const secondPlugin = { name: 'second' };
|
|
27
|
+
registry.register('embedder', 'dup', firstPlugin);
|
|
28
|
+
registry.register('embedder', 'dup', secondPlugin);
|
|
29
|
+
const result = registry.get('embedder', 'dup');
|
|
30
|
+
expect(result).toBe(secondPlugin);
|
|
31
|
+
});
|
|
32
|
+
|
|
33
|
+
test('throws error for unknown plugin type on registration', () => {
|
|
34
|
+
expect(() => registry.register('unknown-type', 'x', {})).toThrow(/Unknown plugin type/);
|
|
35
|
+
});
|
|
36
|
+
|
|
37
|
+
test('throws error when retrieving a nonexistent plugin', () => {
|
|
38
|
+
expect(() => registry.get('retriever', 'missing')).toThrow(/Plugin not found/);
|
|
39
|
+
});
|
|
40
|
+
|
|
41
|
+
test('lists all registered plugin names for a type', () => {
|
|
42
|
+
registry.register('llm', 'gpt-4', {});
|
|
43
|
+
registry.register('llm', 'gpt-3.5', {});
|
|
44
|
+
const list = registry.list('llm');
|
|
45
|
+
expect(list).toContain('gpt-4');
|
|
46
|
+
expect(list).toContain('gpt-3.5');
|
|
47
|
+
expect(list.length).toBe(2);
|
|
48
|
+
});
|
|
49
|
+
|
|
50
|
+
test('throws error when listing plugins for invalid type', () => {
|
|
51
|
+
expect(() => registry.list('invalid')).toThrow(/Unknown plugin type/);
|
|
52
|
+
});
|
|
53
|
+
});
|
|
54
|
+
|
|
55
|
+
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Version: 0.1.0
|
|
3
|
+
* Path: /__tests__/unit/reranker/llm-reranker.test.js
|
|
4
|
+
* Description: Unit tests for LLM-based context reranker
|
|
5
|
+
* Author: Ali Kahwaji
|
|
6
|
+
*/
|
|
7
|
+
|
|
8
|
+
import { LLMReranker } from '../../../src/reranker/llm-reranker.js';
|
|
9
|
+
|
|
10
|
+
describe('LLMReranker', () => {
|
|
11
|
+
const mockDocs = [
|
|
12
|
+
{ text: 'context about math' },
|
|
13
|
+
{ text: 'context about vectors' },
|
|
14
|
+
{ text: 'context about nothing' }
|
|
15
|
+
];
|
|
16
|
+
|
|
17
|
+
test('returns reranked documents based on mock LLM JSON output', async () => {
|
|
18
|
+
const mockLLM = {
|
|
19
|
+
async generate() {
|
|
20
|
+
return '[1, 0, 2]';
|
|
21
|
+
}
|
|
22
|
+
};
|
|
23
|
+
const reranker = new LLMReranker({ llm: mockLLM });
|
|
24
|
+
const result = await reranker.rerank('What are vectors?', mockDocs);
|
|
25
|
+
expect(result.map(d => d.text)).toEqual([
|
|
26
|
+
'context about vectors',
|
|
27
|
+
'context about math',
|
|
28
|
+
'context about nothing'
|
|
29
|
+
]);
|
|
30
|
+
});
|
|
31
|
+
|
|
32
|
+
test('gracefully falls back if LLM returns invalid JSON', async () => {
|
|
33
|
+
const mockLLM = {
|
|
34
|
+
async generate() {
|
|
35
|
+
return 'this is not JSON';
|
|
36
|
+
}
|
|
37
|
+
};
|
|
38
|
+
const reranker = new LLMReranker({ llm: mockLLM });
|
|
39
|
+
const result = await reranker.rerank('Explain.', mockDocs);
|
|
40
|
+
expect(result.length).toBe(0);
|
|
41
|
+
});
|
|
42
|
+
});
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
import { LLMReranker } from '../../../src/reranker/llm-reranker.js';
|
|
2
|
+
|
|
3
|
+
test('LLMReranker output snapshot', async () => {
|
|
4
|
+
const mockLLM = {
|
|
5
|
+
async generate() {
|
|
6
|
+
return '[1, 0, 2]';
|
|
7
|
+
}
|
|
8
|
+
};
|
|
9
|
+
const mockDocs = [
|
|
10
|
+
{ text: 'A' },
|
|
11
|
+
{ text: 'B' },
|
|
12
|
+
{ text: 'C' }
|
|
13
|
+
];
|
|
14
|
+
const reranker = new LLMReranker({ llm: mockLLM });
|
|
15
|
+
const result = await reranker.rerank('What is A?', mockDocs);
|
|
16
|
+
expect(result).toMatchSnapshot();
|
|
17
|
+
});
|
package/bin/cli.js
ADDED
|
@@ -0,0 +1,168 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* Version: 0.1.7
|
|
5
|
+
* Path: /bin/cli.js
|
|
6
|
+
* Description: CLI interface for rag-pipeline utilities with loaders, reranker, and evaluation
|
|
7
|
+
* Author: Ali Kahwaji
|
|
8
|
+
*/
|
|
9
|
+
|
|
10
|
+
import { Command } from 'commander';
|
|
11
|
+
import { createRagPipeline, registry } from '../src/core/create-pipeline.js';
|
|
12
|
+
import { loadRagConfig } from '../src/config/load-config.js';
|
|
13
|
+
import { logger } from '../src/utils/logger.js';
|
|
14
|
+
import { evaluateRagDataset } from '../src/evaluate/evaluator.js';
|
|
15
|
+
import { LLMReranker } from '../src/reranker/llm-reranker.js';
|
|
16
|
+
|
|
17
|
+
import { MarkdownLoader } from '../src/loader/markdown-loader.js';
|
|
18
|
+
import { HTMLLoader } from '../src/loader/html-loader.js';
|
|
19
|
+
import { CSVLoader } from '../src/loader/csv-loader.js';
|
|
20
|
+
import { DirectoryLoader } from '../src/loader/directory-loader.js';
|
|
21
|
+
|
|
22
|
+
// Sample/mock plugin implementations
|
|
23
|
+
class PDFLoader {
|
|
24
|
+
async load(path) {
|
|
25
|
+
return [{ chunk: () => ['Sample PDF chunk.'] }];
|
|
26
|
+
}
|
|
27
|
+
}
|
|
28
|
+
class OpenAIEmbedder {
|
|
29
|
+
async embed(chunks) {
|
|
30
|
+
return chunks.map((text, i) => ({ id: `v${i}`, values: [0.1, 0.2, 0.3], metadata: { text } }));
|
|
31
|
+
}
|
|
32
|
+
async embedQuery(prompt) {
|
|
33
|
+
return [0.1, 0.2, 0.3];
|
|
34
|
+
}
|
|
35
|
+
}
|
|
36
|
+
class PineconeRetriever {
|
|
37
|
+
async store(vectors) {}
|
|
38
|
+
async retrieve(vector) {
|
|
39
|
+
return [
|
|
40
|
+
{ id: 'a', text: 'Chunk about pine trees', metadata: {} },
|
|
41
|
+
{ id: 'b', text: 'Chunk about vectors', metadata: {} },
|
|
42
|
+
{ id: 'c', text: 'Chunk about databases', metadata: {} }
|
|
43
|
+
];
|
|
44
|
+
}
|
|
45
|
+
}
|
|
46
|
+
class OpenAILLM {
|
|
47
|
+
async generate(prompt, context) {
|
|
48
|
+
return `Generated answer using: ${context.map(d => d.text).join(', ')}`;
|
|
49
|
+
}
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
// Register plugins
|
|
53
|
+
registry.register('loader', 'pdf', new PDFLoader());
|
|
54
|
+
registry.register('loader', 'markdown', new MarkdownLoader());
|
|
55
|
+
registry.register('loader', 'html', new HTMLLoader());
|
|
56
|
+
registry.register('loader', 'csv', new CSVLoader());
|
|
57
|
+
registry.register('loader', 'directory', new DirectoryLoader());
|
|
58
|
+
registry.register('embedder', 'openai', new OpenAIEmbedder());
|
|
59
|
+
registry.register('retriever', 'pinecone', new PineconeRetriever());
|
|
60
|
+
registry.register('llm', 'openai-gpt-4', new OpenAILLM());
|
|
61
|
+
|
|
62
|
+
const program = new Command();
|
|
63
|
+
program
|
|
64
|
+
.name('rag-pipeline')
|
|
65
|
+
.description('CLI for RAG pipeline utilities')
|
|
66
|
+
.version('0.1.7');
|
|
67
|
+
|
|
68
|
+
function resolveConfig(cliOpts) {
|
|
69
|
+
try {
|
|
70
|
+
const fileConfig = loadRagConfig();
|
|
71
|
+
return { ...fileConfig, ...cliOpts };
|
|
72
|
+
} catch (err) {
|
|
73
|
+
logger.warn({ error: err.message }, 'Config file fallback failed');
|
|
74
|
+
return cliOpts;
|
|
75
|
+
}
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
program
|
|
79
|
+
.command('ingest')
|
|
80
|
+
.argument('<path>', 'Path to document(s)')
|
|
81
|
+
.option('--loader <type>', 'Document loader type')
|
|
82
|
+
.option('--embedder <type>', 'Embedder type')
|
|
83
|
+
.option('--retriever <type>', 'Vector store retriever type')
|
|
84
|
+
.action(async (path, opts) => {
|
|
85
|
+
try {
|
|
86
|
+
const config = resolveConfig(opts);
|
|
87
|
+
const pipeline = createRagPipeline(config);
|
|
88
|
+
await pipeline.ingest(path);
|
|
89
|
+
logger.info('Ingestion complete');
|
|
90
|
+
} catch (err) {
|
|
91
|
+
logger.error({ error: err.message }, 'Ingestion failed');
|
|
92
|
+
process.exit(1);
|
|
93
|
+
}
|
|
94
|
+
});
|
|
95
|
+
|
|
96
|
+
program
|
|
97
|
+
.command('query')
|
|
98
|
+
.argument('<prompt>', 'Prompt to submit')
|
|
99
|
+
.option('--embedder <type>', 'Embedder type')
|
|
100
|
+
.option('--retriever <type>', 'Retriever type')
|
|
101
|
+
.option('--llm <type>', 'LLM type')
|
|
102
|
+
.action(async (prompt, opts) => {
|
|
103
|
+
try {
|
|
104
|
+
const config = resolveConfig(opts);
|
|
105
|
+
const pipeline = createRagPipeline(config);
|
|
106
|
+
const answer = await pipeline.query(prompt);
|
|
107
|
+
logger.info({ answer }, 'Query complete');
|
|
108
|
+
console.log('\n Answer:\n', answer);
|
|
109
|
+
} catch (err) {
|
|
110
|
+
logger.error({ error: err.message }, 'Query failed');
|
|
111
|
+
process.exit(1);
|
|
112
|
+
}
|
|
113
|
+
});
|
|
114
|
+
|
|
115
|
+
program
|
|
116
|
+
.command('evaluate')
|
|
117
|
+
.argument('<dataset>', 'Path to JSON file of evaluation cases')
|
|
118
|
+
.option('--embedder <type>', 'Embedder type')
|
|
119
|
+
.option('--retriever <type>', 'Retriever type')
|
|
120
|
+
.option('--llm <type>', 'LLM type')
|
|
121
|
+
.action(async (dataset, opts) => {
|
|
122
|
+
try {
|
|
123
|
+
const config = resolveConfig(opts);
|
|
124
|
+
const results = await evaluateRagDataset(dataset, config);
|
|
125
|
+
const passRate = results.filter(r => r.success).length / results.length;
|
|
126
|
+
const avgBLEU = results.reduce((sum, r) => sum + r.scores.bleu, 0) / results.length;
|
|
127
|
+
const avgROUGE = results.reduce((sum, r) => sum + r.scores.rouge, 0) / results.length;
|
|
128
|
+
|
|
129
|
+
results.forEach((r, i) => {
|
|
130
|
+
console.log(`\nCase ${i + 1}:\nPrompt: ${r.prompt}\nPass: ${r.success}\nBLEU: ${r.scores.bleu.toFixed(2)}\nROUGE: ${r.scores.rouge.toFixed(2)}`);
|
|
131
|
+
});
|
|
132
|
+
|
|
133
|
+
console.log(`\n Summary:\nPassed: ${Math.round(passRate * 100)}%\nAvg BLEU: ${avgBLEU.toFixed(2)}\nAvg ROUGE: ${avgROUGE.toFixed(2)}`);
|
|
134
|
+
} catch (err) {
|
|
135
|
+
logger.error({ error: err.message }, 'Evaluation failed');
|
|
136
|
+
process.exit(1);
|
|
137
|
+
}
|
|
138
|
+
});
|
|
139
|
+
|
|
140
|
+
program
|
|
141
|
+
.command('rerank')
|
|
142
|
+
.argument('<prompt>', 'Prompt to rerank context for')
|
|
143
|
+
.option('--retriever <type>', 'Retriever type')
|
|
144
|
+
.option('--llm <type>', 'LLM type')
|
|
145
|
+
.action(async (prompt, opts) => {
|
|
146
|
+
try {
|
|
147
|
+
const config = resolveConfig(opts);
|
|
148
|
+
const retriever = registry.get('retriever', config.retriever);
|
|
149
|
+
const llm = registry.get('llm', config.llm);
|
|
150
|
+
const embedder = registry.get('embedder', config.embedder);
|
|
151
|
+
|
|
152
|
+
const queryVector = await embedder.embedQuery(prompt);
|
|
153
|
+
const docs = await retriever.retrieve(queryVector);
|
|
154
|
+
|
|
155
|
+
const reranker = new LLMReranker({ llm });
|
|
156
|
+
const reranked = await reranker.rerank(prompt, docs);
|
|
157
|
+
|
|
158
|
+
console.log('\n Reranked Results:');
|
|
159
|
+
reranked.forEach((doc, i) => {
|
|
160
|
+
console.log(`\n#${i + 1}: ${doc.text}`);
|
|
161
|
+
});
|
|
162
|
+
} catch (err) {
|
|
163
|
+
logger.error({ error: err.message }, 'Rerank failed');
|
|
164
|
+
process.exit(1);
|
|
165
|
+
}
|
|
166
|
+
});
|
|
167
|
+
|
|
168
|
+
program.parse();
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# Website
|
|
2
|
+
|
|
3
|
+
This website is built using [Docusaurus](https://docusaurus.io/), a modern static website generator.
|
|
4
|
+
|
|
5
|
+
### Installation
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
$ yarn
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
### Local Development
|
|
12
|
+
|
|
13
|
+
```
|
|
14
|
+
$ yarn start
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.
|
|
18
|
+
|
|
19
|
+
### Build
|
|
20
|
+
|
|
21
|
+
```
|
|
22
|
+
$ yarn build
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
This command generates static content into the `build` directory and can be served using any static contents hosting service.
|
|
26
|
+
|
|
27
|
+
### Deployment
|
|
28
|
+
|
|
29
|
+
Using SSH:
|
|
30
|
+
|
|
31
|
+
```
|
|
32
|
+
$ USE_SSH=true yarn deploy
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Not using SSH:
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
$ GIT_USER=<Your GitHub username> yarn deploy
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch.
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
---
|
|
2
|
+
slug: first-blog-post
|
|
3
|
+
title: First Blog Post
|
|
4
|
+
authors: [slorber, yangshun]
|
|
5
|
+
tags: [hola, docusaurus]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
Lorem ipsum dolor sit amet...
|
|
9
|
+
|
|
10
|
+
<!-- truncate -->
|
|
11
|
+
|
|
12
|
+
...consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
---
|
|
2
|
+
slug: long-blog-post
|
|
3
|
+
title: Long Blog Post
|
|
4
|
+
authors: yangshun
|
|
5
|
+
tags: [hello, docusaurus]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
This is the summary of a very long blog post,
|
|
9
|
+
|
|
10
|
+
Use a `<!--` `truncate` `-->` comment to limit blog post size in the list view.
|
|
11
|
+
|
|
12
|
+
<!-- truncate -->
|
|
13
|
+
|
|
14
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
15
|
+
|
|
16
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
17
|
+
|
|
18
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
19
|
+
|
|
20
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
21
|
+
|
|
22
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
23
|
+
|
|
24
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
25
|
+
|
|
26
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
27
|
+
|
|
28
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
29
|
+
|
|
30
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
31
|
+
|
|
32
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
33
|
+
|
|
34
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
35
|
+
|
|
36
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
37
|
+
|
|
38
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
39
|
+
|
|
40
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
41
|
+
|
|
42
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
43
|
+
|
|
44
|
+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
---
|
|
2
|
+
slug: mdx-blog-post
|
|
3
|
+
title: MDX Blog Post
|
|
4
|
+
authors: [slorber]
|
|
5
|
+
tags: [docusaurus]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
Blog posts support [Docusaurus Markdown features](https://docusaurus.io/docs/markdown-features), such as [MDX](https://mdxjs.com/).
|
|
9
|
+
|
|
10
|
+
:::tip
|
|
11
|
+
|
|
12
|
+
Use the power of React to create interactive blog posts.
|
|
13
|
+
|
|
14
|
+
:::
|
|
15
|
+
|
|
16
|
+
{/* truncate */}
|
|
17
|
+
|
|
18
|
+
For example, use JSX to create an interactive button:
|
|
19
|
+
|
|
20
|
+
```js
|
|
21
|
+
<button onClick={() => alert('button clicked!')}>Click me!</button>
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
<button onClick={() => alert('button clicked!')}>Click me!</button>
|
|
Binary file
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
---
|
|
2
|
+
slug: welcome
|
|
3
|
+
title: Welcome
|
|
4
|
+
authors: [slorber, yangshun]
|
|
5
|
+
tags: [facebook, hello, docusaurus]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
[Docusaurus blogging features](https://docusaurus.io/docs/blog) are powered by the [blog plugin](https://docusaurus.io/docs/api/plugins/@docusaurus/plugin-content-blog).
|
|
9
|
+
|
|
10
|
+
Here are a few tips you might find useful.
|
|
11
|
+
|
|
12
|
+
<!-- truncate -->
|
|
13
|
+
|
|
14
|
+
Simply add Markdown files (or folders) to the `blog` directory.
|
|
15
|
+
|
|
16
|
+
Regular blog authors can be added to `authors.yml`.
|
|
17
|
+
|
|
18
|
+
The blog post date can be extracted from filenames, such as:
|
|
19
|
+
|
|
20
|
+
- `2019-05-30-welcome.md`
|
|
21
|
+
- `2019-05-30-welcome/index.md`
|
|
22
|
+
|
|
23
|
+
A blog post folder can be convenient to co-locate blog post images:
|
|
24
|
+
|
|
25
|
+

|
|
26
|
+
|
|
27
|
+
The blog supports tags as well!
|
|
28
|
+
|
|
29
|
+
**And if you don't want a blog**: just delete this directory, and use `blog: false` in your Docusaurus config.
|