doc2vec 1.0.0 → 1.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (21) hide show
  1. package/README.md +21 -0
  2. package/config.yaml.new +239 -165
  3. package/config.yaml.ok +164 -0
  4. package/config.yaml.solo +251 -0
  5. package/dist/doc2vec.js +164 -84
  6. package/dist/solo-reference-architectures/Gloo_Edge/ALB_Last_Mile_Encryption/data/acm-waf-cdk/lib/acm-cdk-stack.js +20 -0
  7. package/dist/solo-reference-architectures/Gloo_Edge/ALB_Last_Mile_Encryption/data/acm-waf-cdk/lib/route53-cdk-stack.js +33 -0
  8. package/dist/solo-reference-architectures/Gloo_Edge/ALB_Last_Mile_Encryption/data/acm-waf-cdk/lib/waf-cdk-stack.js +66 -0
  9. package/dist/solo-reference-architectures/Gloo_Edge/ALB_Last_Mile_Encryption/data/acm-waf-cdk/test/s3-cloudfront-cdk.test.js +16 -0
  10. package/dist/solo-reference-architectures/Gloo_Edge/Cloudfront_NLB/data/s3-cloudfront-cdk/lib/acm-cdk-stack.js +21 -0
  11. package/dist/solo-reference-architectures/Gloo_Edge/Cloudfront_NLB/data/s3-cloudfront-cdk/lib/s3-cloudfront-cdk-stack.js +95 -0
  12. package/dist/solo-reference-architectures/Gloo_Edge/Cloudfront_NLB/data/s3-cloudfront-cdk/test/s3-cloudfront-cdk.test.js +16 -0
  13. package/dist/solo-reference-architectures/Gloo_Gateway/aws-nlb-tls-offloading/data/acm-cdk/bin/acm-cdk.js +55 -0
  14. package/dist/solo-reference-architectures/Gloo_Gateway/aws-nlb-tls-offloading/data/acm-cdk/lib/acm-cdk-stack.js +20 -0
  15. package/dist/solo-reference-architectures/Gloo_Gateway/aws-nlb-tls-offloading/data/acm-cdk/lib/route53-cdk-stack.js +33 -0
  16. package/dist/solo-reference-architectures/Gloo_Gateway/aws-nlb-tls-offloading/data/acm-cdk/test/s3-cloudfront-cdk.test.js +16 -0
  17. package/doc2vec.ts +184 -93
  18. package/doc2vec.ts.ok +1730 -0
  19. package/package.json +1 -1
  20. package/tmp/README.md.old +1 -0
  21. package/tmp.db +0 -0
package/README.md CHANGED
@@ -1,5 +1,7 @@
1
1
  # Doc2Vec
2
2
 
3
+ [![npm version](https://img.shields.io/npm/v/doc2vec.svg)](https://www.npmjs.com/package/doc2vec)
4
+
3
5
  This project provides a configurable tool (`doc2vec`) to crawl specified websites (typically documentation sites), GitHub repositories, and local directories, extract relevant content, convert it to Markdown, chunk it intelligently, generate vector embeddings using OpenAI, and store the chunks along with their embeddings in a vector database (SQLite with `sqlite-vec` or Qdrant).
4
6
 
5
7
  The primary goal is to prepare documentation content for Retrieval-Augmented Generation (RAG) systems or semantic search applications.
@@ -87,6 +89,7 @@ Configuration is managed through two files:
87
89
  * `include_extensions`: (Optional) Array of file extensions to include (e.g., `['.md', '.txt']`). Defaults to `['.md', '.txt', '.html', '.htm']`.
88
90
  * `exclude_extensions`: (Optional) Array of file extensions to exclude.
89
91
  * `recursive`: (Optional) Whether to traverse subdirectories (defaults to `true`).
92
+ * `url_rewrite_prefix` (Optional) URL prefix to rewrite `file://` URLs (e.g., `https://mydomain.com`)
90
93
  * `encoding`: (Optional) File encoding to use (defaults to `'utf8'`).
91
94
 
92
95
  Common configuration for all types:
@@ -195,6 +198,24 @@ The script will then:
195
198
  * Uses `@qdrant/js-client-rest`.
196
199
  * Requires `qdrant_url`, `qdrant_port`, `collection_name` and potentially `QDRANT_API_KEY`.
197
200
 
201
+ ## Now Available via npx
202
+
203
+ You can run `doc2vec` without cloning the repo or installing it globally. Just use:
204
+
205
+ ```bash
206
+ npx doc2vec [path/to/your/config.yaml]
207
+ ```
208
+
209
+ This will:
210
+
211
+ 1. Fetch the latest version of doc2vec from npm.
212
+
213
+ 2. Load and process the sources defined in your config.yaml.
214
+
215
+ 3. Generate, embed, and store documentation chunks in the configured database(s).
216
+
217
+ If you don’t specify a config path, it will look for config.yaml in the current working directory.
218
+
198
219
  ## Core Logic Flow
199
220
 
200
221
  1. **Load Config:** Read and parse `config.yaml`.
package/config.yaml.new CHANGED
@@ -1,179 +1,253 @@
1
1
  # Doc2Vec Configuration
2
2
  sources:
3
+ # Local directory Sources
3
4
  - type: local_directory
4
5
  product_name: 'solo-reference-architectures'
5
6
  version: 'latest'
6
- path: '/home/denis/Downloads/solo-reference-architectures'
7
+ path: './solo-reference-architectures'
7
8
  include_extensions: ['.md']
8
9
  exclude_extensions: []
9
10
  recursive: true
11
+ url_rewrite_prefix: https://github.com/solo-io/solo-reference-architectures/blob/main
10
12
  max_size: 1048576
11
13
  database_config:
12
14
  type: 'qdrant'
13
15
  params:
14
16
  qdrant_url: 'https://qdrant.is.solo.io'
15
17
  qdrant_port: 443
16
- collection_name: 'tmp'
17
-
18
- # # GitHub Sources
19
- # - type: github
20
- # product_name: 'istio'
21
- # version: 'latest'
22
- # repo: 'istio/istio'
23
- # start_date: '2025-01-01'
24
- # max_size: 1048576
25
- # database_config:
26
- # type: 'sqlite'
27
- # params:
28
- # db_path: './istio-issues.db'
29
-
30
- # # Website Sources
31
- # - type: website
32
- # product_name: 'ambient'
33
- # version: 'latest'
34
- # url: 'https://ambientmesh.io/docs/'
35
- # max_size: 1048576
36
- # database_config:
37
- # type: 'sqlite'
38
- # params:
39
- # db_path: './ambient.db'
40
-
41
- # - type: website
42
- # product_name: 'argo'
43
- # version: 'latest'
44
- # url: 'https://argo-cd.readthedocs.io/en/stable/'
45
- # max_size: 1048576
46
- # database_config:
47
- # type: 'sqlite'
48
- # params:
49
- # db_path: './argo.db'
50
-
51
- # - type: website
52
- # product_name: 'argo-rollouts'
53
- # version: 'latest'
54
- # url: 'https://argoproj.github.io/argo-rollouts/installation/'
55
- # max_size: 1048576
56
- # database_config:
57
- # type: 'sqlite'
58
- # params:
59
- # db_path: './argo-rollouts.db'
60
-
61
- # - type: website
62
- # product_name: 'argo-rollouts'
63
- # version: 'latest'
64
- # url: 'https://rollouts-plugin-trafficrouter-gatewayapi.readthedocs.io/en/latest/'
65
- # max_size: 1048576
66
- # database_config:
67
- # type: 'sqlite'
68
- # params:
69
- # db_path: './argo-rollouts.db'
70
-
71
- # - type: website
72
- # product_name: 'cilium'
73
- # version: 'latest'
74
- # url: 'https://docs.cilium.io/en/stable/'
75
- # max_size: 1048576
76
- # database_config:
77
- # type: 'sqlite'
78
- # params:
79
- # db_path: './cilium.db'
80
-
81
- # - type: website
82
- # product_name: 'gateway-api'
83
- # version: 'latest'
84
- # url: 'https://gateway-api.sigs.k8s.io/'
85
- # max_size: 1048576
86
- # database_config:
87
- # type: 'sqlite'
88
- # params:
89
- # db_path: './gateway-api.db'
90
-
91
- # - type: website
92
- # product_name: 'gloo-mesh-core'
93
- # version: 'latest'
94
- # url: 'https://docs.solo.io/gloo-mesh/latest/'
95
- # max_size: 1048576
96
- # database_config:
97
- # type: 'sqlite'
98
- # params:
99
- # db_path: './gloo-mesh-core.db'
100
-
101
- # - type: website
102
- # product_name: 'gloo-mesh-enterprise'
103
- # version: 'latest'
104
- # url: 'https://docs.solo.io/gloo-mesh-enterprise/latest/'
105
- # max_size: 1048576
106
- # database_config:
107
- # type: 'sqlite'
108
- # params:
109
- # db_path: './gloo-mesh-enterprise.db'
110
-
111
- # - type: website
112
- # product_name: 'gloo-edge'
113
- # version: 'latest'
114
- # url: 'https://docs.solo.io/gloo-edge/latest/'
115
- # max_size: 1048576
116
- # database_config:
117
- # type: 'sqlite'
118
- # params:
119
- # db_path: './gloo-edge.db'
120
-
121
- # - type: website
122
- # product_name: 'gloo-gateway'
123
- # version: 'latest'
124
- # url: 'https://docs.solo.io/gateway/latest/'
125
- # max_size: 1048576
126
- # database_config:
127
- # type: 'sqlite'
128
- # params:
129
- # db_path: './gloo-gateway.db'
130
-
131
- # - type: website
132
- # product_name: 'helm'
133
- # version: 'latest'
134
- # url: 'https://helm.sh/docs/'
135
- # max_size: 1048576
136
- # database_config:
137
- # type: 'sqlite'
138
- # params:
139
- # db_path: './helm.db'
140
-
141
- # - type: website
142
- # product_name: 'istio'
143
- # version: 'latest'
144
- # url: 'https://istio.io/latest/docs/'
145
- # max_size: 1048576
146
- # database_config:
147
- # type: 'sqlite'
148
- # params:
149
- # db_path: './istio.db'
150
-
151
- # - type: website
152
- # product_name: 'kubernetes'
153
- # version: 'latest'
154
- # url: 'https://kubernetes.io/docs/'
155
- # max_size: 1048576
156
- # database_config:
157
- # type: 'sqlite'
158
- # params:
159
- # db_path: './kubernetes.db'
160
-
161
- # - type: website
162
- # product_name: 'otel'
163
- # version: 'latest'
164
- # url: 'https://opentelemetry.io/docs/'
165
- # max_size: 1048576
166
- # database_config:
167
- # type: 'sqlite'
168
- # params:
169
- # db_path: './otel.db'
170
-
171
- # - type: website
172
- # product_name: 'prometheus'
173
- # version: 'latest'
174
- # url: 'https://prometheus.io/docs/'
175
- # max_size: 1048576
176
- # database_config:
177
- # type: 'sqlite'
178
- # params:
179
- # db_path: './prometheus.db'
18
+ collection_name: 'github-solo-reference-architectures'
19
+
20
+ # GitHub Sources
21
+ - type: github
22
+ product_name: 'istio'
23
+ version: 'latest'
24
+ repo: 'istio/istio'
25
+ start_date: '2024-01-01'
26
+ max_size: 1048576
27
+ database_config:
28
+ type: 'qdrant'
29
+ params:
30
+ qdrant_url: 'https://qdrant.is.solo.io'
31
+ qdrant_port: 443
32
+ collection_name: 'github-istio'
33
+
34
+ # GitHub Sources
35
+ - type: github
36
+ product_name: 'istio'
37
+ version: 'latest'
38
+ repo: 'istio/istio'
39
+ start_date: '2024-01-01'
40
+ max_size: 1048576
41
+ database_config:
42
+ type: 'qdrant'
43
+ params:
44
+ qdrant_url: 'https://qdrant.is.solo.io'
45
+ qdrant_port: 443
46
+ collection_name: 'github-istio'
47
+
48
+ - type: github
49
+ product_name: 'solo-projects'
50
+ version: 'latest'
51
+ repo: 'solo-io/solo-projects'
52
+ start_date: '2024-01-01'
53
+ max_size: 1048576
54
+ database_config:
55
+ type: 'qdrant'
56
+ params:
57
+ qdrant_url: 'https://qdrant.is.solo.io'
58
+ qdrant_port: 443
59
+ collection_name: 'github-solo-projects'
60
+
61
+ - type: github
62
+ product_name: 'gloo-mesh-enterprise'
63
+ version: 'latest'
64
+ repo: 'solo-io/gloo-mesh-enterprise'
65
+ start_date: '2024-01-01'
66
+ max_size: 1048576
67
+ database_config:
68
+ type: 'qdrant'
69
+ params:
70
+ qdrant_url: 'https://qdrant.is.solo.io'
71
+ qdrant_port: 443
72
+ collection_name: 'github-gloo-mesh-enterprise'
73
+
74
+ # Website Sources
75
+ - type: website
76
+ product_name: 'ambient'
77
+ version: 'latest'
78
+ url: 'https://ambientmesh.io/docs/'
79
+ max_size: 1048576
80
+ database_config:
81
+ type: 'qdrant'
82
+ params:
83
+ qdrant_url: 'https://qdrant.is.solo.io'
84
+ qdrant_port: 443
85
+ collection_name: 'ambient'
86
+
87
+ - type: website
88
+ product_name: 'argo'
89
+ version: 'latest'
90
+ url: 'https://argo-cd.readthedocs.io/en/stable/'
91
+ max_size: 1048576
92
+ database_config:
93
+ type: 'qdrant'
94
+ params:
95
+ qdrant_url: 'https://qdrant.is.solo.io'
96
+ qdrant_port: 443
97
+ collection_name: 'argo'
98
+
99
+ - type: website
100
+ product_name: 'argo-rollouts'
101
+ version: 'latest'
102
+ url: 'https://argoproj.github.io/argo-rollouts/installation/'
103
+ max_size: 1048576
104
+ database_config:
105
+ type: 'qdrant'
106
+ params:
107
+ qdrant_url: 'https://qdrant.is.solo.io'
108
+ qdrant_port: 443
109
+ collection_name: 'argo-rollouts'
110
+
111
+ - type: website
112
+ product_name: 'argo-rollouts'
113
+ version: 'latest'
114
+ url: 'https://rollouts-plugin-trafficrouter-gatewayapi.readthedocs.io/en/latest/'
115
+ max_size: 1048576
116
+ database_config:
117
+ type: 'qdrant'
118
+ params:
119
+ qdrant_url: 'https://qdrant.is.solo.io'
120
+ qdrant_port: 443
121
+ collection_name: 'argo-rollouts'
122
+
123
+ - type: website
124
+ product_name: 'cilium'
125
+ version: 'latest'
126
+ url: 'https://docs.cilium.io/en/stable/'
127
+ max_size: 1048576
128
+ database_config:
129
+ type: 'qdrant'
130
+ params:
131
+ qdrant_url: 'https://qdrant.is.solo.io'
132
+ qdrant_port: 443
133
+ collection_name: 'cilium'
134
+
135
+ - type: website
136
+ product_name: 'gateway-api'
137
+ version: 'latest'
138
+ url: 'https://gateway-api.sigs.k8s.io/'
139
+ max_size: 1048576
140
+ database_config:
141
+ type: 'qdrant'
142
+ params:
143
+ qdrant_url: 'https://qdrant.is.solo.io'
144
+ qdrant_port: 443
145
+ collection_name: 'gateway-api'
146
+
147
+ - type: website
148
+ product_name: 'gloo-mesh-core'
149
+ version: 'latest'
150
+ url: 'https://docs.solo.io/gloo-mesh/latest/'
151
+ max_size: 1048576
152
+ database_config:
153
+ type: 'qdrant'
154
+ params:
155
+ qdrant_url: 'https://qdrant.is.solo.io'
156
+ qdrant_port: 443
157
+ collection_name: 'gloo-mesh-core'
158
+
159
+ - type: website
160
+ product_name: 'gloo-mesh-enterprise'
161
+ version: 'latest'
162
+ url: 'https://docs.solo.io/gloo-mesh-enterprise/latest/'
163
+ max_size: 1048576
164
+ database_config:
165
+ type: 'qdrant'
166
+ params:
167
+ qdrant_url: 'https://qdrant.is.solo.io'
168
+ qdrant_port: 443
169
+ collection_name: 'gloo-mesh-enterprise'
170
+
171
+ - type: website
172
+ product_name: 'gloo-edge'
173
+ version: 'latest'
174
+ url: 'https://docs.solo.io/gloo-edge/latest/'
175
+ max_size: 1048576
176
+ database_config:
177
+ type: 'qdrant'
178
+ params:
179
+ qdrant_url: 'https://qdrant.is.solo.io'
180
+ qdrant_port: 443
181
+ collection_name: 'gloo-edge'
182
+
183
+ - type: website
184
+ product_name: 'gloo-gateway'
185
+ version: 'latest'
186
+ url: 'https://docs.solo.io/gateway/latest/'
187
+ max_size: 1048576
188
+ database_config:
189
+ type: 'qdrant'
190
+ params:
191
+ qdrant_url: 'https://qdrant.is.solo.io'
192
+ qdrant_port: 443
193
+ collection_name: 'gloo-gateway'
194
+
195
+ - type: website
196
+ product_name: 'helm'
197
+ version: 'latest'
198
+ url: 'https://helm.sh/docs/'
199
+ max_size: 1048576
200
+ database_config:
201
+ type: 'qdrant'
202
+ params:
203
+ qdrant_url: 'https://qdrant.is.solo.io'
204
+ qdrant_port: 443
205
+ collection_name: 'helm'
206
+
207
+ - type: website
208
+ product_name: 'istio'
209
+ version: 'latest'
210
+ url: 'https://istio.io/latest/docs/'
211
+ max_size: 1048576
212
+ database_config:
213
+ type: 'qdrant'
214
+ params:
215
+ qdrant_url: 'https://qdrant.is.solo.io'
216
+ qdrant_port: 443
217
+ collection_name: 'istio'
218
+
219
+ - type: website
220
+ product_name: 'kubernetes'
221
+ version: 'latest'
222
+ url: 'https://kubernetes.io/docs/'
223
+ max_size: 1048576
224
+ database_config:
225
+ type: 'qdrant'
226
+ params:
227
+ qdrant_url: 'https://qdrant.is.solo.io'
228
+ qdrant_port: 443
229
+ collection_name: 'kubernetes'
230
+
231
+ - type: website
232
+ product_name: 'otel'
233
+ version: 'latest'
234
+ url: 'https://opentelemetry.io/docs/'
235
+ max_size: 1048576
236
+ database_config:
237
+ type: 'qdrant'
238
+ params:
239
+ qdrant_url: 'https://qdrant.is.solo.io'
240
+ qdrant_port: 443
241
+ collection_name: 'otel'
242
+
243
+ - type: website
244
+ product_name: 'prometheus'
245
+ version: 'latest'
246
+ url: 'https://prometheus.io/docs/'
247
+ max_size: 1048576
248
+ database_config:
249
+ type: 'qdrant'
250
+ params:
251
+ qdrant_url: 'https://qdrant.is.solo.io'
252
+ qdrant_port: 443
253
+ collection_name: 'prometheus'
package/config.yaml.ok ADDED
@@ -0,0 +1,164 @@
1
+ # Doc2Vec Configuration
2
+ sources:
3
+ # GitHub Sources
4
+ - type: github
5
+ product_name: 'istio'
6
+ version: 'latest'
7
+ repo: 'istio/istio'
8
+ start_date: '2025-01-01'
9
+ max_size: 1048576
10
+ database_config:
11
+ type: 'sqlite'
12
+ params:
13
+ db_path: './istio-issues.db'
14
+
15
+ # Website Sources
16
+ - type: website
17
+ product_name: 'ambient'
18
+ version: 'latest'
19
+ url: 'https://ambientmesh.io/docs/'
20
+ max_size: 1048576
21
+ database_config:
22
+ type: 'sqlite'
23
+ params:
24
+ db_path: './ambient.db'
25
+
26
+ - type: website
27
+ product_name: 'argo'
28
+ version: 'latest'
29
+ url: 'https://argo-cd.readthedocs.io/en/stable/'
30
+ max_size: 1048576
31
+ database_config:
32
+ type: 'sqlite'
33
+ params:
34
+ db_path: './argo.db'
35
+
36
+ - type: website
37
+ product_name: 'argo-rollouts'
38
+ version: 'latest'
39
+ url: 'https://argoproj.github.io/argo-rollouts/installation/'
40
+ max_size: 1048576
41
+ database_config:
42
+ type: 'sqlite'
43
+ params:
44
+ db_path: './argo-rollouts.db'
45
+
46
+ - type: website
47
+ product_name: 'argo-rollouts'
48
+ version: 'latest'
49
+ url: 'https://rollouts-plugin-trafficrouter-gatewayapi.readthedocs.io/en/latest/'
50
+ max_size: 1048576
51
+ database_config:
52
+ type: 'sqlite'
53
+ params:
54
+ db_path: './argo-rollouts.db'
55
+
56
+ - type: website
57
+ product_name: 'cilium'
58
+ version: 'latest'
59
+ url: 'https://docs.cilium.io/en/stable/'
60
+ max_size: 1048576
61
+ database_config:
62
+ type: 'sqlite'
63
+ params:
64
+ db_path: './cilium.db'
65
+
66
+ - type: website
67
+ product_name: 'gateway-api'
68
+ version: 'latest'
69
+ url: 'https://gateway-api.sigs.k8s.io/'
70
+ max_size: 1048576
71
+ database_config:
72
+ type: 'sqlite'
73
+ params:
74
+ db_path: './gateway-api.db'
75
+
76
+ - type: website
77
+ product_name: 'gloo-mesh-core'
78
+ version: 'latest'
79
+ url: 'https://docs.solo.io/gloo-mesh/latest/'
80
+ max_size: 1048576
81
+ database_config:
82
+ type: 'sqlite'
83
+ params:
84
+ db_path: './gloo-mesh-core.db'
85
+
86
+ - type: website
87
+ product_name: 'gloo-mesh-enterprise'
88
+ version: 'latest'
89
+ url: 'https://docs.solo.io/gloo-mesh-enterprise/latest/'
90
+ max_size: 1048576
91
+ database_config:
92
+ type: 'sqlite'
93
+ params:
94
+ db_path: './gloo-mesh-enterprise.db'
95
+
96
+ - type: website
97
+ product_name: 'gloo-edge'
98
+ version: 'latest'
99
+ url: 'https://docs.solo.io/gloo-edge/latest/'
100
+ max_size: 1048576
101
+ database_config:
102
+ type: 'sqlite'
103
+ params:
104
+ db_path: './gloo-edge.db'
105
+
106
+ - type: website
107
+ product_name: 'gloo-gateway'
108
+ version: 'latest'
109
+ url: 'https://docs.solo.io/gateway/latest/'
110
+ max_size: 1048576
111
+ database_config:
112
+ type: 'sqlite'
113
+ params:
114
+ db_path: './gloo-gateway.db'
115
+
116
+ - type: website
117
+ product_name: 'helm'
118
+ version: 'latest'
119
+ url: 'https://helm.sh/docs/'
120
+ max_size: 1048576
121
+ database_config:
122
+ type: 'sqlite'
123
+ params:
124
+ db_path: './helm.db'
125
+
126
+ - type: website
127
+ product_name: 'istio'
128
+ version: 'latest'
129
+ url: 'https://istio.io/latest/docs/'
130
+ max_size: 1048576
131
+ database_config:
132
+ type: 'sqlite'
133
+ params:
134
+ db_path: './istio.db'
135
+
136
+ - type: website
137
+ product_name: 'kubernetes'
138
+ version: 'latest'
139
+ url: 'https://kubernetes.io/docs/'
140
+ max_size: 1048576
141
+ database_config:
142
+ type: 'sqlite'
143
+ params:
144
+ db_path: './kubernetes.db'
145
+
146
+ - type: website
147
+ product_name: 'otel'
148
+ version: 'latest'
149
+ url: 'https://opentelemetry.io/docs/'
150
+ max_size: 1048576
151
+ database_config:
152
+ type: 'sqlite'
153
+ params:
154
+ db_path: './otel.db'
155
+
156
+ - type: website
157
+ product_name: 'prometheus'
158
+ version: 'latest'
159
+ url: 'https://prometheus.io/docs/'
160
+ max_size: 1048576
161
+ database_config:
162
+ type: 'sqlite'
163
+ params:
164
+ db_path: './prometheus.db'