pgsemantic 0.2.0__tar.gz → 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/PKG-INFO +53 -1
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/README.md +52 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/__init__.py +1 -1
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/commands/search.py +57 -8
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/db/vectors.py +3 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/web/app.py +8 -18
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/web/static/index.html +6 -4
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic.egg-info/PKG-INFO +53 -1
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pyproject.toml +1 -1
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/LICENSE +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/__main__.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/cli.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/commands/__init__.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/commands/apply.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/commands/index.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/commands/inspect.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/commands/integrate.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/commands/serve.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/commands/status.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/commands/ui.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/commands/worker.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/config.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/db/__init__.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/db/client.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/db/introspect.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/db/queue.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/embeddings/__init__.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/embeddings/base.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/embeddings/local.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/embeddings/ollama_provider.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/embeddings/openai_provider.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/exceptions.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/mcp_server/__init__.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/mcp_server/server.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/web/__init__.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/worker/__init__.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic/worker/daemon.py +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic.egg-info/SOURCES.txt +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic.egg-info/dependency_links.txt +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic.egg-info/entry_points.txt +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic.egg-info/requires.txt +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/pgsemantic.egg-info/top_level.txt +0 -0
- {pgsemantic-0.2.0 → pgsemantic-0.3.0}/setup.cfg +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: pgsemantic
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.3.0
|
|
4
4
|
Summary: Zero-config semantic search bootstrap for any PostgreSQL database
|
|
5
5
|
Author: Sai Ram Varma Budharaju
|
|
6
6
|
License-Expression: MIT
|
|
@@ -49,6 +49,58 @@ It works on your existing tables, keeps embeddings in sync automatically as rows
|
|
|
49
49
|
|
|
50
50
|
---
|
|
51
51
|
|
|
52
|
+
## See it in action
|
|
53
|
+
|
|
54
|
+
We loaded 5,980 real clinical trials from ClinicalTrials.gov and searched them in plain English. Here's what the full flow looks like.
|
|
55
|
+
|
|
56
|
+
**Browse your existing tables — no setup required**
|
|
57
|
+
|
|
58
|
+
<img src="docs/screenshots/01-tables.png" width="100%" alt="Table Browser showing clinical_trials and news_articles tables with row counts"/>
|
|
59
|
+
|
|
60
|
+
*pgsemantic connects to your existing database and shows every table. Nothing is modified until you say so.*
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
**Inspect scores your text columns for semantic search suitability**
|
|
65
|
+
|
|
66
|
+
<img src="docs/screenshots/02-inspect.png" width="100%" alt="Inspect page scoring clinical_trials columns — eligibility and brief_summary rated 3 stars"/>
|
|
67
|
+
|
|
68
|
+
*Columns are ranked by average text length and content quality. `brief_summary` and `eligibility` score highest — these are the best candidates for embedding.*
|
|
69
|
+
|
|
70
|
+
---
|
|
71
|
+
|
|
72
|
+
**Apply sets up everything with one click**
|
|
73
|
+
|
|
74
|
+
<img src="docs/screenshots/03-apply.png" width="100%" alt="Apply page with column picker checkboxes and embedding model selection"/>
|
|
75
|
+
|
|
76
|
+
*Pick your columns, choose an embedding model (local runs free on your machine, no API key), click Apply Setup. pgsemantic adds a vector column, HNSW index, and a change-tracking trigger.*
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
**Dashboard shows your embedding coverage**
|
|
81
|
+
|
|
82
|
+
<img src="docs/screenshots/04-dashboard.png" width="100%" alt="Dashboard showing 5,980 embeddings, 100% coverage, 0 pending"/>
|
|
83
|
+
|
|
84
|
+
*5,980 rows. 100% coverage. 0 pending. Every clinical trial is now searchable by meaning.*
|
|
85
|
+
|
|
86
|
+
---
|
|
87
|
+
|
|
88
|
+
**Search in plain English — results that share zero words with your query**
|
|
89
|
+
|
|
90
|
+
<img src="docs/screenshots/05-search.png" width="100%" alt="Search results for 'kidney complications in diabetics' returning trials about diabetic nephropathy and DKD"/>
|
|
91
|
+
|
|
92
|
+
*Query: `kidney complications in diabetics`*
|
|
93
|
+
|
|
94
|
+
*Top results: trials about **diabetic nephropathy**, **DKD**, **renal insufficiency** — none of which contain the words "kidney complications in diabetics". That's semantic search.*
|
|
95
|
+
|
|
96
|
+
Results export to CSV or JSON directly from the UI, or from the CLI:
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
pgsemantic search "cardiovascular risk factors" --table clinical_trials --format csv > results.csv
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
---
|
|
103
|
+
|
|
52
104
|
## What is semantic search?
|
|
53
105
|
|
|
54
106
|
Traditional SQL search finds rows where a column contains the exact words you typed. Semantic search understands *meaning* — it finds results that are conceptually similar even when they share no words with your query.
|
|
@@ -12,6 +12,58 @@ It works on your existing tables, keeps embeddings in sync automatically as rows
|
|
|
12
12
|
|
|
13
13
|
---
|
|
14
14
|
|
|
15
|
+
## See it in action
|
|
16
|
+
|
|
17
|
+
We loaded 5,980 real clinical trials from ClinicalTrials.gov and searched them in plain English. Here's what the full flow looks like.
|
|
18
|
+
|
|
19
|
+
**Browse your existing tables — no setup required**
|
|
20
|
+
|
|
21
|
+
<img src="docs/screenshots/01-tables.png" width="100%" alt="Table Browser showing clinical_trials and news_articles tables with row counts"/>
|
|
22
|
+
|
|
23
|
+
*pgsemantic connects to your existing database and shows every table. Nothing is modified until you say so.*
|
|
24
|
+
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
**Inspect scores your text columns for semantic search suitability**
|
|
28
|
+
|
|
29
|
+
<img src="docs/screenshots/02-inspect.png" width="100%" alt="Inspect page scoring clinical_trials columns — eligibility and brief_summary rated 3 stars"/>
|
|
30
|
+
|
|
31
|
+
*Columns are ranked by average text length and content quality. `brief_summary` and `eligibility` score highest — these are the best candidates for embedding.*
|
|
32
|
+
|
|
33
|
+
---
|
|
34
|
+
|
|
35
|
+
**Apply sets up everything with one click**
|
|
36
|
+
|
|
37
|
+
<img src="docs/screenshots/03-apply.png" width="100%" alt="Apply page with column picker checkboxes and embedding model selection"/>
|
|
38
|
+
|
|
39
|
+
*Pick your columns, choose an embedding model (local runs free on your machine, no API key), click Apply Setup. pgsemantic adds a vector column, HNSW index, and a change-tracking trigger.*
|
|
40
|
+
|
|
41
|
+
---
|
|
42
|
+
|
|
43
|
+
**Dashboard shows your embedding coverage**
|
|
44
|
+
|
|
45
|
+
<img src="docs/screenshots/04-dashboard.png" width="100%" alt="Dashboard showing 5,980 embeddings, 100% coverage, 0 pending"/>
|
|
46
|
+
|
|
47
|
+
*5,980 rows. 100% coverage. 0 pending. Every clinical trial is now searchable by meaning.*
|
|
48
|
+
|
|
49
|
+
---
|
|
50
|
+
|
|
51
|
+
**Search in plain English — results that share zero words with your query**
|
|
52
|
+
|
|
53
|
+
<img src="docs/screenshots/05-search.png" width="100%" alt="Search results for 'kidney complications in diabetics' returning trials about diabetic nephropathy and DKD"/>
|
|
54
|
+
|
|
55
|
+
*Query: `kidney complications in diabetics`*
|
|
56
|
+
|
|
57
|
+
*Top results: trials about **diabetic nephropathy**, **DKD**, **renal insufficiency** — none of which contain the words "kidney complications in diabetics". That's semantic search.*
|
|
58
|
+
|
|
59
|
+
Results export to CSV or JSON directly from the UI, or from the CLI:
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
pgsemantic search "cardiovascular risk factors" --table clinical_trials --format csv > results.csv
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
---
|
|
66
|
+
|
|
15
67
|
## What is semantic search?
|
|
16
68
|
|
|
17
69
|
Traditional SQL search finds rows where a column contains the exact words you typed. Semantic search understands *meaning* — it finds results that are conceptually similar even when they share no words with your query.
|
|
@@ -1,9 +1,13 @@
|
|
|
1
1
|
"""pgsemantic search — semantic search from the command line."""
|
|
2
2
|
from __future__ import annotations
|
|
3
3
|
|
|
4
|
+
import csv
|
|
5
|
+
import json
|
|
6
|
+
import sys
|
|
7
|
+
from typing import Any
|
|
8
|
+
|
|
4
9
|
import typer
|
|
5
10
|
from rich.console import Console
|
|
6
|
-
from rich.panel import Panel
|
|
7
11
|
|
|
8
12
|
from pgsemantic.config import load_project_config, load_settings
|
|
9
13
|
from pgsemantic.db.client import get_connection
|
|
@@ -13,7 +17,28 @@ from pgsemantic.embeddings import get_provider
|
|
|
13
17
|
console = Console()
|
|
14
18
|
|
|
15
19
|
# Columns to hide from search results (internal/large)
|
|
16
|
-
|
|
20
|
+
_TABLE_HIDDEN_COLUMNS = frozenset({"embedding", "similarity", "content"})
|
|
21
|
+
|
|
22
|
+
|
|
23
|
+
def _results_to_csv(results: list[dict[str, Any]]) -> None:
|
|
24
|
+
"""Write results as CSV to stdout."""
|
|
25
|
+
if not results:
|
|
26
|
+
return
|
|
27
|
+
writer = csv.DictWriter(sys.stdout, fieldnames=list(results[0].keys()))
|
|
28
|
+
writer.writeheader()
|
|
29
|
+
for row in results:
|
|
30
|
+
writer.writerow({k: str(v) if v is not None else "" for k, v in row.items()})
|
|
31
|
+
|
|
32
|
+
|
|
33
|
+
def _results_to_json(results: list[dict[str, Any]]) -> None:
|
|
34
|
+
"""Write results as a JSON array to stdout."""
|
|
35
|
+
print(json.dumps(results, indent=2, default=str))
|
|
36
|
+
|
|
37
|
+
|
|
38
|
+
def _results_to_jsonl(results: list[dict[str, Any]]) -> None:
|
|
39
|
+
"""Write results as newline-delimited JSON to stdout."""
|
|
40
|
+
for row in results:
|
|
41
|
+
print(json.dumps(row, default=str))
|
|
17
42
|
|
|
18
43
|
|
|
19
44
|
def search_command(
|
|
@@ -39,8 +64,18 @@ def search_command(
|
|
|
39
64
|
"-d",
|
|
40
65
|
help="Database URL (overrides DATABASE_URL env var).",
|
|
41
66
|
),
|
|
67
|
+
fmt: str = typer.Option(
|
|
68
|
+
"table",
|
|
69
|
+
"--format",
|
|
70
|
+
"-f",
|
|
71
|
+
help="Output format: table (default), csv, json, jsonl.",
|
|
72
|
+
),
|
|
42
73
|
) -> None:
|
|
43
74
|
"""Search your database using natural language."""
|
|
75
|
+
if fmt not in ("table", "csv", "json", "jsonl"):
|
|
76
|
+
console.print("[red]--format must be one of: table, csv, json, jsonl[/red]")
|
|
77
|
+
raise typer.Exit(code=1)
|
|
78
|
+
|
|
44
79
|
settings = load_settings()
|
|
45
80
|
database_url = db if db else settings.database_url
|
|
46
81
|
|
|
@@ -88,9 +123,25 @@ def search_command(
|
|
|
88
123
|
)
|
|
89
124
|
|
|
90
125
|
if not results:
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
126
|
+
if fmt == "table":
|
|
127
|
+
console.print("\n[yellow]No results found.[/yellow]\n")
|
|
128
|
+
elif fmt == "json":
|
|
129
|
+
print("[]")
|
|
130
|
+
# csv and jsonl: empty output is correct
|
|
131
|
+
raise typer.Exit(code=0) # re-raised by outer except typer.Exit guard
|
|
132
|
+
|
|
133
|
+
# --- machine-readable output ---
|
|
134
|
+
if fmt == "csv":
|
|
135
|
+
_results_to_csv(results)
|
|
136
|
+
return
|
|
137
|
+
if fmt == "json":
|
|
138
|
+
_results_to_json(results)
|
|
139
|
+
return
|
|
140
|
+
if fmt == "jsonl":
|
|
141
|
+
_results_to_jsonl(results)
|
|
142
|
+
return
|
|
143
|
+
|
|
144
|
+
# --- default: Rich table output ---
|
|
94
145
|
console.print()
|
|
95
146
|
console.print(
|
|
96
147
|
f"[dim]Results for:[/dim] [cyan]\"{query}\"[/cyan] "
|
|
@@ -101,15 +152,13 @@ def search_command(
|
|
|
101
152
|
sim = float(str(row["similarity"]))
|
|
102
153
|
score_color = "green" if sim >= 0.5 else "yellow" if sim >= 0.3 else "dim"
|
|
103
154
|
|
|
104
|
-
# Build metadata line from non-hidden, non-content columns
|
|
105
155
|
meta_parts: list[str] = []
|
|
106
156
|
for key, val in row.items():
|
|
107
|
-
if key in
|
|
157
|
+
if key in _TABLE_HIDDEN_COLUMNS or key == column:
|
|
108
158
|
continue
|
|
109
159
|
meta_parts.append(f"[dim]{key}:[/dim] {val}")
|
|
110
160
|
meta_line = " ".join(meta_parts)
|
|
111
161
|
|
|
112
|
-
# Truncate content for preview
|
|
113
162
|
content = str(row.get(column, ""))
|
|
114
163
|
if len(content) > 300:
|
|
115
164
|
content = content[:300] + "..."
|
|
@@ -625,6 +625,9 @@ def search_similar(
|
|
|
625
625
|
quoted_col = _quoted(column)
|
|
626
626
|
sql = SQL_SEARCH_SIMILAR.format(table=qualified, column=quoted_col)
|
|
627
627
|
|
|
628
|
+
# ef_search must be >= limit or HNSW returns fewer results than requested.
|
|
629
|
+
ef_search = max(limit * 2, 100)
|
|
630
|
+
conn.execute(f"SET hnsw.ef_search = {ef_search}")
|
|
628
631
|
result = conn.execute(
|
|
629
632
|
sql, {"query_vector": query_vector, "limit": limit}
|
|
630
633
|
).fetchall()
|
|
@@ -719,7 +719,7 @@ async def bulk_apply(req: BulkApplyRequest):
|
|
|
719
719
|
|
|
720
720
|
|
|
721
721
|
@app.post("/api/index")
|
|
722
|
-
|
|
722
|
+
def index_table(req: IndexRequest):
|
|
723
723
|
"""Bulk embed existing rows."""
|
|
724
724
|
db_url = _get_db_url()
|
|
725
725
|
settings = load_settings()
|
|
@@ -737,15 +737,10 @@ async def index_table(req: IndexRequest):
|
|
|
737
737
|
provider = get_provider(table_config.model, api_key=api_key)
|
|
738
738
|
|
|
739
739
|
with get_connection(db_url) as conn:
|
|
740
|
-
|
|
741
|
-
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
else:
|
|
745
|
-
total = count_total_with_content(
|
|
746
|
-
conn, req.table, table_config.column, table_config.schema,
|
|
747
|
-
source_columns=table_config.source_columns,
|
|
748
|
-
)
|
|
740
|
+
total = count_total_with_content(
|
|
741
|
+
conn, req.table, table_config.column, table_config.schema,
|
|
742
|
+
source_columns=table_config.source_columns,
|
|
743
|
+
)
|
|
749
744
|
already = count_embedded(
|
|
750
745
|
conn, req.table, table_config.schema,
|
|
751
746
|
storage_mode=table_config.storage_mode,
|
|
@@ -814,7 +809,7 @@ async def index_table(req: IndexRequest):
|
|
|
814
809
|
|
|
815
810
|
|
|
816
811
|
@app.post("/api/reindex")
|
|
817
|
-
|
|
812
|
+
def reindex_table(req: IndexRequest):
|
|
818
813
|
"""Re-embed a table by nulling existing embeddings then re-indexing."""
|
|
819
814
|
db_url = _get_db_url()
|
|
820
815
|
|
|
@@ -837,7 +832,7 @@ async def reindex_table(req: IndexRequest):
|
|
|
837
832
|
conn.commit()
|
|
838
833
|
|
|
839
834
|
# Now run normal index
|
|
840
|
-
return
|
|
835
|
+
return index_table(req)
|
|
841
836
|
|
|
842
837
|
except HTTPException:
|
|
843
838
|
raise
|
|
@@ -1001,12 +996,7 @@ async def status_dashboard():
|
|
|
1001
996
|
for tc in config.tables:
|
|
1002
997
|
try:
|
|
1003
998
|
embedded = count_embedded(conn, tc.table, schema=tc.schema, storage_mode=tc.storage_mode, shadow_table=tc.shadow_table)
|
|
1004
|
-
|
|
1005
|
-
shadow_qualified = f'"{tc.schema}"."{tc.shadow_table}"' if tc.schema else f'"{tc.shadow_table}"'
|
|
1006
|
-
row = conn.execute(f"SELECT COUNT(*) AS cnt FROM {shadow_qualified}").fetchone()
|
|
1007
|
-
total = int(str(row["cnt"])) if row else 0
|
|
1008
|
-
else:
|
|
1009
|
-
total = count_total_with_content(conn, tc.table, tc.column, schema=tc.schema, source_columns=tc.source_columns)
|
|
999
|
+
total = count_total_with_content(conn, tc.table, tc.column, schema=tc.schema, source_columns=tc.source_columns)
|
|
1010
1000
|
pending = count_pending(conn, tc.table)
|
|
1011
1001
|
failed = count_failed(conn, tc.table)
|
|
1012
1002
|
pct = min(round(embedded / total * 100, 1), 100.0) if total > 0 else 0.0
|
|
@@ -373,7 +373,7 @@
|
|
|
373
373
|
<button type="submit" class="btn btn-primary" id="btn-search">Search</button>
|
|
374
374
|
</form>
|
|
375
375
|
</div></div>
|
|
376
|
-
<div id="s-results" class="hidden"><div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:12px"><span id="s-title" style="font-weight:600;color:var(--fg-0);font-size:14px"></span><span id="s-count" class="pill pill-blue"></span></div><div id="s-output"></div></div>
|
|
376
|
+
<div id="s-results" class="hidden"><div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:12px"><span id="s-title" style="font-weight:600;color:var(--fg-0);font-size:14px"></span><div style="display:flex;align-items:center;gap:8px"><span id="s-count" class="pill pill-blue"></span><div id="s-download" class="hidden" style="display:flex;gap:6px"><button class="btn btn-ghost btn-sm" onclick="downloadSearchCSV()">Download CSV</button><button class="btn btn-ghost btn-sm" onclick="downloadSearchJSON()">Download JSON</button></div></div></div><div id="s-output"></div></div>
|
|
377
377
|
</div>
|
|
378
378
|
|
|
379
379
|
<!-- INSPECT -->
|
|
@@ -583,8 +583,10 @@ function saveSearchHistory(t,q){if(!searchHistory[t])searchHistory[t]=[];searchH
|
|
|
583
583
|
$('s-query')?.addEventListener?.('input',()=>{clearTimeout(searchTimer);searchTimer=setTimeout(()=>{if($('s-query').value.trim().length>=3&&$('s-table').value)doSearch();},300);});
|
|
584
584
|
$('s-table')?.addEventListener?.('change',loadSearchHistory);
|
|
585
585
|
let _searchResults=[];
|
|
586
|
-
|
|
587
|
-
|
|
586
|
+
function downloadSearchCSV(){if(!_searchResults.length)return;const keys=Object.keys(_searchResults[0]);const rows=[keys.map(k=>'"'+k.replace(/"/g,'""')+'"').join(','),..._searchResults.map(r=>keys.map(k=>{const v=r[k]===null||r[k]===undefined?'':String(r[k]);return'"'+v.replace(/"/g,'""')+'"';}).join(','))];const blob=new Blob([rows.join('\n')],{type:'text/csv'});const a=document.createElement('a');a.href=URL.createObjectURL(blob);a.download=`pgsemantic_search_${Date.now()}.csv`;a.click();setTimeout(()=>URL.revokeObjectURL(a.href),100);}
|
|
587
|
+
function downloadSearchJSON(){if(!_searchResults.length)return;const blob=new Blob([JSON.stringify(_searchResults,null,2)],{type:'application/json'});const a=document.createElement('a');a.href=URL.createObjectURL(blob);a.download=`pgsemantic_search_${Date.now()}.json`;a.click();setTimeout(()=>URL.revokeObjectURL(a.href),100);}
|
|
588
|
+
async function doSearch(e){if(e)e.preventDefault();const table=$('s-table').value,query=$('s-query').value.trim(),limit=parseInt($('s-limit').value)||5;if(!table||!query)return;const ctrl=startOp('search');setBtn('btn-search',true,'search');$('s-results').classList.remove('hidden');$('s-download').classList.add('hidden');$('s-output').innerHTML='<div class="loading"><span class="spin"></span>Searching...</div>';try{const d=await api('/api/search',{method:'POST',body:JSON.stringify({query,table,limit}),signal:ctrl.signal});saveSearchHistory(table,query);loadSearchHistory();$('s-title').textContent=`Results for "${query}"`;$('s-count').textContent=`${(d.results||[]).length} results`;if(!(d.results||[]).length){$('s-download').classList.add('hidden');$('s-output').innerHTML='<div class="empty-state"><div class="empty-state-icon">🔎</div><h3>No results found</h3><p>Try different keywords or check that the table is indexed.</p></div>';return;}_searchResults=d.results;$('s-download').classList.remove('hidden');let h='';for(let i=0;i<d.results.length;i++){const r=d.results[i],sim=r.similarity||0;const sc=sim>=0.5?'score-high':sim>=0.3?'score-mid':'score-low';let meta=[];for(const[k,v]of Object.entries(r)){if(k==='similarity'||k==='content')continue;meta.push(`<span style="color:var(--fg-4)">${esc(k)}:</span> ${esc(String(v))}`);}const barColor=sim>=0.5?'var(--green)':sim>=0.3?'var(--amber)':'var(--fg-4)';const barWidth=Math.round(Math.min(sim,1)*100);
|
|
589
|
+
h+=`<div class="result" onclick="openDrawer(buildRowDrawer(_searchResults[${i}]))"><div class="result-head"><span class="result-rank ${sc}">#${i+1}<span class="result-score-bar-wrap"><span class="result-score-bar" style="width:${barWidth}%;background:${barColor}"></span></span>${sim.toFixed(4)}</span><span class="result-meta">${meta.join(' ')}</span></div><div class="result-body">${esc(r.content||'')}</div></div>`;}$('s-output').innerHTML=h;}catch(e){if(e.name==='AbortError'){$('s-download').classList.add('hidden');$('s-output').innerHTML='<p style="color:var(--fg-4)">Search cancelled.</p>';}else{$('s-download').classList.add('hidden');$('s-output').innerHTML=`<p style="color:var(--red)">${esc(e.message)}</p>`;toast(e.message,false);}}finally{finishOp('search');setBtn('btn-search',false,'search');}}
|
|
588
590
|
function buildRowDrawer(row){let h='<div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:16px"><h3>Row Details</h3><button class="btn btn-ghost btn-sm" onclick="closeDrawer()">✕</button></div>';for(const[k,v]of Object.entries(row)){if(k==='similarity'){h+=`<div class="kv-row"><div class="kv-key">Similarity</div><div class="kv-val" style="font-weight:600;font-family:var(--mono)">${Number(v).toFixed(4)}</div></div>`;continue;}h+=`<div class="kv-row"><div class="kv-key">${esc(k)}</div><div class="kv-val">${esc(String(v||''))}</div></div>`;}return h;}
|
|
589
591
|
|
|
590
592
|
// Inspect
|
|
@@ -604,7 +606,7 @@ $('a-table')?.addEventListener?.('change',function(){const val=this.value;const
|
|
|
604
606
|
async function loadApplyTables(){const sel=$('a-table');if(!sel)return;try{const d=await api('/api/database/tables');const tables=d.tables||[];const cur=sel.value;sel.innerHTML='<option value="">Select a table...</option>';for(const t of tables){const o=document.createElement('option');o.value=t.name;o.textContent=t.name;o.dataset.schema=t.schema||'public';sel.appendChild(o);}if(cur){sel.value=cur;}else{sel.value='';}}catch(e){sel.innerHTML='<option value="">Could not load tables</option>';}}
|
|
605
607
|
let _pendingApplyBody=null;
|
|
606
608
|
async function doApply(e){e.preventDefault();const table=$('a-table').value.trim(),col=$('a-col').value.trim(),cols=$('a-cols').value.trim();const model=$('a-model').value,ext=$('a-ext').checked,schema=$('a-schema').value.trim()||'public';if(!col&&!cols){toast('Specify a column',false);return;}if(col&&cols){toast('Use column OR columns, not both',false);return;}_pendingApplyBody=JSON.stringify({table,column:col,columns:cols,model,external:ext,schema_name:schema});openModal(`<h3>Confirm Apply</h3><div class="dialog-text">This will modify your database:<ul style="margin:8px 0 0 16px"><li>Add an embedding column (or shadow table) to <strong>${esc(table)}</strong></li><li>Create an HNSW index for similarity search</li><li>Install a change-tracking trigger</li></ul></div><div class="dialog-actions"><button class="btn btn-secondary" onclick="closeModal()">Cancel</button><button class="btn btn-primary" onclick="closeModal();execApply()">Confirm</button></div>`);}
|
|
607
|
-
async function execApply(){const bodyStr=_pendingApplyBody;const ctrl=startOp('apply');setBtn('btn-apply',true,'apply');$('a-results').classList.remove('hidden');$('a-results').innerHTML='<div class="card"><div class="card-body"><div class="loading"><span class="spin"></span>Setting up...</div></div></div>';try{const d=await api('/api/apply',{method:'POST',body:bodyStr,signal:ctrl.signal});let h='<div class="card"><div class="card-head"><h3>Setup Complete</h3></div><div class="card-body">';if(d.sql_preview)h+=`<div class="code-block" style="margin-bottom:16px">${esc(d.sql_preview)}</div>`;for(const s of(d.steps||[])){const ok=s.status==='ok'||s.status==='created';const cls=ok?'step-ok':s.status==='already_exists'?'step-skip':'step-fail';const icon=ok?'✓':s.status==='already_exists'?'—':'✗';h+=`<div class="step-row"><span class="step-icon ${cls}">${icon}</span><span>${esc(s.step)}</span><span class="pill ${ok?'pill-green':'pill-amber'}">${esc(s.status)}${s.version?' v'+esc(s.version):''}</span></div>`;}h+=`</div><div class="card-foot"><button class="btn btn-primary btn-sm" onclick="go('index')">Index this table
|
|
609
|
+
async function execApply(){const bodyStr=_pendingApplyBody;const ctrl=startOp('apply');setBtn('btn-apply',true,'apply');$('a-results').classList.remove('hidden');$('a-results').innerHTML='<div class="card"><div class="card-body"><div class="loading"><span class="spin"></span>Setting up...</div></div></div>';try{const d=await api('/api/apply',{method:'POST',body:bodyStr,signal:ctrl.signal});let h='<div class="card"><div class="card-head"><h3>Setup Complete</h3></div><div class="card-body">';if(d.sql_preview)h+=`<div class="code-block" style="margin-bottom:16px">${esc(d.sql_preview)}</div>`;for(const s of(d.steps||[])){const ok=s.status==='ok'||s.status==='created';const cls=ok?'step-ok':s.status==='already_exists'?'step-skip':'step-fail';const icon=ok?'✓':s.status==='already_exists'?'—':'✗';h+=`<div class="step-row"><span class="step-icon ${cls}">${icon}</span><span>${esc(s.step)}</span><span class="pill ${ok?'pill-green':'pill-amber'}">${esc(s.status)}${s.version?' v'+esc(s.version):''}</span></div>`;}h+=`</div><div class="card-foot"><button class="btn btn-primary btn-sm" onclick="go('index');$('x-table').value='${esc(d.table)}'">Index this table →</button><button class="btn btn-secondary btn-sm" onclick="go('dashboard')">Dashboard</button></div></div>`;$('a-results').innerHTML=h;toast('Setup complete — index your rows to enable search');checkConn();}catch(e){if(e.name==='AbortError')$('a-results').innerHTML='<div class="card"><div class="card-body"><p style="color:var(--fg-4)">Setup cancelled.</p></div></div>';else{$('a-results').innerHTML=`<div class="card"><div class="card-body"><p style="color:var(--red)">${esc(e.message)}</p></div></div>`;toast(e.message,false);}}finally{finishOp('apply');setBtn('btn-apply',false,'apply');}}
|
|
608
610
|
|
|
609
611
|
// Index
|
|
610
612
|
let _indexPollTimer=null;
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: pgsemantic
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.3.0
|
|
4
4
|
Summary: Zero-config semantic search bootstrap for any PostgreSQL database
|
|
5
5
|
Author: Sai Ram Varma Budharaju
|
|
6
6
|
License-Expression: MIT
|
|
@@ -49,6 +49,58 @@ It works on your existing tables, keeps embeddings in sync automatically as rows
|
|
|
49
49
|
|
|
50
50
|
---
|
|
51
51
|
|
|
52
|
+
## See it in action
|
|
53
|
+
|
|
54
|
+
We loaded 5,980 real clinical trials from ClinicalTrials.gov and searched them in plain English. Here's what the full flow looks like.
|
|
55
|
+
|
|
56
|
+
**Browse your existing tables — no setup required**
|
|
57
|
+
|
|
58
|
+
<img src="docs/screenshots/01-tables.png" width="100%" alt="Table Browser showing clinical_trials and news_articles tables with row counts"/>
|
|
59
|
+
|
|
60
|
+
*pgsemantic connects to your existing database and shows every table. Nothing is modified until you say so.*
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
**Inspect scores your text columns for semantic search suitability**
|
|
65
|
+
|
|
66
|
+
<img src="docs/screenshots/02-inspect.png" width="100%" alt="Inspect page scoring clinical_trials columns — eligibility and brief_summary rated 3 stars"/>
|
|
67
|
+
|
|
68
|
+
*Columns are ranked by average text length and content quality. `brief_summary` and `eligibility` score highest — these are the best candidates for embedding.*
|
|
69
|
+
|
|
70
|
+
---
|
|
71
|
+
|
|
72
|
+
**Apply sets up everything with one click**
|
|
73
|
+
|
|
74
|
+
<img src="docs/screenshots/03-apply.png" width="100%" alt="Apply page with column picker checkboxes and embedding model selection"/>
|
|
75
|
+
|
|
76
|
+
*Pick your columns, choose an embedding model (local runs free on your machine, no API key), click Apply Setup. pgsemantic adds a vector column, HNSW index, and a change-tracking trigger.*
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
**Dashboard shows your embedding coverage**
|
|
81
|
+
|
|
82
|
+
<img src="docs/screenshots/04-dashboard.png" width="100%" alt="Dashboard showing 5,980 embeddings, 100% coverage, 0 pending"/>
|
|
83
|
+
|
|
84
|
+
*5,980 rows. 100% coverage. 0 pending. Every clinical trial is now searchable by meaning.*
|
|
85
|
+
|
|
86
|
+
---
|
|
87
|
+
|
|
88
|
+
**Search in plain English — results that share zero words with your query**
|
|
89
|
+
|
|
90
|
+
<img src="docs/screenshots/05-search.png" width="100%" alt="Search results for 'kidney complications in diabetics' returning trials about diabetic nephropathy and DKD"/>
|
|
91
|
+
|
|
92
|
+
*Query: `kidney complications in diabetics`*
|
|
93
|
+
|
|
94
|
+
*Top results: trials about **diabetic nephropathy**, **DKD**, **renal insufficiency** — none of which contain the words "kidney complications in diabetics". That's semantic search.*
|
|
95
|
+
|
|
96
|
+
Results export to CSV or JSON directly from the UI, or from the CLI:
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
pgsemantic search "cardiovascular risk factors" --table clinical_trials --format csv > results.csv
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
---
|
|
103
|
+
|
|
52
104
|
## What is semantic search?
|
|
53
105
|
|
|
54
106
|
Traditional SQL search finds rows where a column contains the exact words you typed. Semantic search understands *meaning* — it finds results that are conceptually similar even when they share no words with your query.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|