folderlake 0.2.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
folderlake/viewer.html ADDED
@@ -0,0 +1,397 @@
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head>
4
+ <meta charset="utf-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1">
6
+ <title>FolderLake</title>
7
+ <style>
8
+ :root { --bg:#f4f4f4; --ink:#111; --muted:#666; --line:#ddd; --accent:#ed7d31; --blue:#4a80d0; }
9
+ * { box-sizing:border-box; }
10
+ body { margin:0; font-family:system-ui,-apple-system,"Segoe UI",sans-serif; background:var(--bg); color:var(--ink); height:100vh; display:flex; flex-direction:column; }
11
+ header { display:flex; gap:12px; align-items:center; padding:10px 16px; border-bottom:1px solid var(--line); background:#fff; }
12
+ header h1 { font-size:18px; margin:0; }
13
+ header .folder { color:var(--muted); font-size:13px; overflow:hidden; text-overflow:ellipsis; white-space:nowrap; }
14
+ header input[type=search] { margin-left:auto; padding:6px 10px; font-size:14px; width:280px; border:1px solid var(--line); border-radius:6px; }
15
+ .view-toggle button, .edge-toggles label { cursor:pointer; }
16
+ .view-toggle button { padding:6px 12px; border:1px solid var(--line); background:#fff; border-radius:6px; font-size:13px; }
17
+ .view-toggle button.active { background:var(--ink); color:#fff; border-color:var(--ink); }
18
+ main { flex:1; display:grid; grid-template-columns:250px 320px minmax(0,1fr) 380px; min-height:0; }
19
+ @media (max-width: 1280px) {
20
+ main { grid-template-columns:230px 280px minmax(0,1fr) 320px; }
21
+ }
22
+ @media (min-width: 921px) {
23
+ /* Desktop shows list and graph side by side — no view toggle. */
24
+ .view-toggle { display:none; }
25
+ }
26
+ aside, section { overflow-y:auto; padding:12px 14px; }
27
+ aside { border-right:1px solid var(--line); background:#fff; }
28
+ aside h2, section h2 { font-size:12px; text-transform:uppercase; letter-spacing:.06em; color:var(--muted); margin:14px 0 6px; }
29
+ aside h2:first-child { margin-top:0; }
30
+ .kind-row, .topic-row { display:flex; justify-content:space-between; gap:8px; padding:3px 0; font-size:14px; cursor:pointer; }
31
+ .topic-row.active { font-weight:700; color:var(--accent); }
32
+ .tree-row { display:flex; justify-content:space-between; gap:8px; padding:3px 0; font-size:14px; cursor:pointer; align-items:baseline; }
33
+ .tree-row.active { font-weight:700; color:var(--accent); }
34
+ .tree-row .tbody { flex:1; min-width:0; display:flex; align-items:baseline; }
35
+ .tree-row .tlabel { flex:1; min-width:0; overflow:hidden; text-overflow:ellipsis; white-space:nowrap; }
36
+ .tree-row .count { flex:none; }
37
+ .tree-toggle { display:inline-block; width:14px; color:var(--muted); cursor:pointer; flex:none; }
38
+ .count { color:var(--muted); font-size:12px; }
39
+ .clear-btn { margin-top:10px; padding:5px 10px; font-size:13px; border:1px solid var(--line); background:#fff; border-radius:6px; cursor:pointer; }
40
+ #list-pane { border-right:1px solid var(--line); padding:0; }
41
+ #graph-pane { border-right:1px solid var(--line); padding:0; display:flex; flex-direction:column; overflow:hidden; background:#fff; }
42
+ #graph-wrap { flex:1; display:flex; flex-direction:column; min-height:0; padding:8px; }
43
+ #file-list { list-style:none; margin:0; padding:0; }
44
+ #file-list li { padding:10px 14px; border-bottom:1px solid var(--line); cursor:pointer; }
45
+ #file-list li:hover { background:#fafafa; }
46
+ #file-list li.selected { background:#fff3e8; border-left:3px solid var(--accent); }
47
+ #file-list .fname { font-size:14px; font-weight:600; word-break:break-all; }
48
+ #file-list .fmeta { font-size:12px; color:var(--muted); margin-top:2px; }
49
+ .tag { display:inline-block; font-size:11px; background:#eef3fb; color:var(--blue); border-radius:10px; padding:1px 8px; margin:2px 2px 0 0; }
50
+ #graph-chart { width:100%; flex:1; min-height:220px; }
51
+ .graph-actions { padding:4px; }
52
+ .graph-actions a { font-size:13px; color:var(--blue); text-decoration:none; border:1px solid var(--line); background:#fff; border-radius:6px; padding:5px 10px; }
53
+ .graph-actions a:hover { border-color:var(--blue); }
54
+ #edge-info { font-size:13px; color:var(--muted); padding:6px 4px; min-height:30px; }
55
+ .edge-toggles { font-size:13px; display:flex; gap:12px; flex-wrap:wrap; padding:4px; }
56
+ #preview-pane { background:#fff; }
57
+ #preview-pane h2 { font-size:16px; text-transform:none; letter-spacing:0; color:var(--ink); margin:0 0 4px; word-break:break-all; }
58
+ #preview-pane .meta { font-size:12px; color:var(--muted); line-height:1.6; word-break:break-all; }
59
+ #preview-pane .summary { font-size:14px; line-height:1.5; margin:12px 0; }
60
+ #preview-pane pre { white-space:pre-wrap; word-break:break-word; font-size:13px; line-height:1.5; background:var(--bg); padding:10px; border-radius:6px; max-height:45vh; overflow-y:auto; }
61
+ #preview-pane table { border-collapse:collapse; font-size:13px; margin:8px 0; }
62
+ #preview-pane th, #preview-pane td { border:1px solid var(--line); padding:4px 8px; text-align:left; }
63
+ #preview-pane code { font-size:12px; background:var(--bg); padding:2px 6px; border-radius:4px; word-break:break-all; }
64
+ .copy-btn { padding:4px 10px; font-size:12px; border:1px solid var(--line); background:#fff; border-radius:6px; cursor:pointer; margin-left:6px; }
65
+ .placeholder { color:var(--muted); font-size:14px; margin-top:40px; text-align:center; }
66
+ .hidden { display:none !important; }
67
+ /* Small screens: the desktop layout is three fixed columns
68
+ (250px + fluid + 400px) which cannot fit a phone — the file list
69
+ collapses to zero width. Stack the panes vertically instead. */
70
+ @media (max-width: 920px) {
71
+ body { height:auto; }
72
+ header { flex-wrap:wrap; }
73
+ header .folder { flex:1; min-width:0; }
74
+ header input[type=search] { order:4; width:100%; margin-left:0; }
75
+ main { display:flex; flex-direction:column; }
76
+ aside, section { overflow:visible; }
77
+ aside { border-right:none; border-bottom:1px solid var(--line);
78
+ max-height:50vh; overflow-y:auto; }
79
+ #list-pane { border-right:none; }
80
+ #graph-pane { border-right:none; border-top:1px solid var(--line); }
81
+ #preview-pane { border-top:1px solid var(--line); }
82
+ #preview-pane pre { max-height:60vh; }
83
+ #graph-chart { height:65vh; flex:none; }
84
+ }
85
+ </style>
86
+ </head>
87
+ <body>
88
+ <header>
89
+ <h1>FolderLake</h1>
90
+ <span class="folder" id="folder-label"></span>
91
+ <input type="search" id="search" placeholder="Search files… (name, summary, text)">
92
+ <div class="view-toggle">
93
+ <button id="btn-list" class="active">List</button>
94
+ <button id="btn-graph">Graph</button>
95
+ </div>
96
+ </header>
97
+ <main>
98
+ <aside id="filters">
99
+ <h2>Type</h2>
100
+ <div id="kind-filters"></div>
101
+ <h2>Topics</h2>
102
+ <div id="topic-filters"></div>
103
+ <button class="clear-btn" id="clear-filters">Clear filters</button>
104
+ </aside>
105
+ <section id="list-pane">
106
+ <ul id="file-list"></ul>
107
+ </section>
108
+ <section id="graph-pane">
109
+ <div id="graph-wrap">
110
+ <div class="edge-toggles" id="edge-toggles"></div>
111
+ <div class="graph-actions"><a href="/api/export.cypher" id="cypher-link" download>Download Cypher (Neo4j)</a></div>
112
+ <div id="edge-info">Drag to pan, scroll to zoom. Click a node to preview it. Click an edge to see why two files are linked.</div>
113
+ <div id="graph-chart"></div>
114
+ </div>
115
+ </section>
116
+ <section id="preview-pane">
117
+ <p class="placeholder">Select a file to preview it here.</p>
118
+ </section>
119
+ </main>
120
+ <script src="/assets/echarts.min.js"></script>
121
+ <script>
122
+ const $ = (s) => document.querySelector(s);
123
+ const state = { kinds: new Set(), topic: null, node: null, expanded: new Set(), q: "", view: "graph",
124
+ files: [], selected: null, graph: null, chart: null, nodeIndex: null,
125
+ edgeKinds: new Set(["shared_topic", "shared_entity", "reference"]) };
126
+ const KIND_COLORS = { text: "#10b070", document: "#4a80d0", data: "#ed7d31", other: "#999" };
127
+ const EDGE_COLORS = { shared_topic: "#4a80d0", shared_entity: "#10b070", reference: "#ed7d31", same_folder: "#ccc" };
128
+ const esc = (s) => String(s ?? "").replace(/[&<>"]/g, (c) => ({"&":"&amp;","<":"&lt;",">":"&gt;",'"':"&quot;"}[c]));
129
+ const fmtSize = (n) => n < 1024 ? n + " B" : n < 1048576 ? (n / 1024).toFixed(1) + " KB" : (n / 1048576).toFixed(2) + " MB";
130
+ const fmtDate = (t) => t ? new Date(t * 1000).toLocaleDateString() : "";
131
+
132
+ async function api(path) { const r = await fetch(path); if (!r.ok) throw new Error(await r.text()); return r.json(); }
133
+
134
+ function renderTopicTree(roots) {
135
+ const rows = [];
136
+ const walk = (node) => {
137
+ const hasKids = node.children && node.children.length > 0;
138
+ const open = state.expanded.has(node.id);
139
+ rows.push(`<div class="tree-row ${state.node === node.id ? "active" : ""}" data-node="${esc(node.id)}" title="${esc(node.label)}" style="padding-left:${node.depth * 14}px">` +
140
+ `<span class="tbody"><span class="tree-toggle" data-toggle="${esc(node.id)}">${hasKids ? (open ? "▾" : "▸") : "·"}</span><span class="tlabel">${esc(node.label)}</span></span>` +
141
+ `<span class="count">${node.count}</span></div>`);
142
+ if (hasKids && open) node.children.forEach(walk);
143
+ };
144
+ roots.forEach(walk);
145
+ $("#topic-filters").innerHTML = rows.join("");
146
+ document.querySelectorAll(".tree-toggle").forEach(tg => tg.addEventListener("click", (ev) => {
147
+ ev.stopPropagation();
148
+ const id = tg.dataset.toggle;
149
+ state.expanded.has(id) ? state.expanded.delete(id) : state.expanded.add(id);
150
+ renderTopicTree(roots);
151
+ }));
152
+ document.querySelectorAll(".tree-row").forEach(row => row.addEventListener("click", () => {
153
+ state.node = state.node === row.dataset.node ? null : row.dataset.node;
154
+ renderTopicTree(roots);
155
+ loadFiles();
156
+ }));
157
+ }
158
+
159
+ async function loadFacets() {
160
+ const f = await api("/api/facets");
161
+ $("#kind-filters").innerHTML = f.kinds.map(k =>
162
+ `<label class="kind-row"><span><input type="checkbox" data-kind="${esc(k.kind)}"> ${esc(k.kind)}</span><span class="count">${k.count}</span></label>`).join("");
163
+ document.querySelectorAll("#kind-filters input").forEach(cb => cb.addEventListener("change", () => {
164
+ cb.checked ? state.kinds.add(cb.dataset.kind) : state.kinds.delete(cb.dataset.kind);
165
+ loadFiles();
166
+ }));
167
+ let tree = { roots: [] };
168
+ try { tree = await api("/api/topics"); } catch (e) { /* older catalog */ }
169
+ if (tree.roots && tree.roots.length) {
170
+ tree.roots.forEach(r => state.expanded.add(r.id));
171
+ renderTopicTree(tree.roots);
172
+ } else {
173
+ $("#topic-filters").innerHTML = f.topics.map(t =>
174
+ `<div class="topic-row" data-topic="${esc(t.tag)}"><span>${esc(t.tag)}</span><span class="count">${t.count}</span></div>`).join("");
175
+ document.querySelectorAll(".topic-row").forEach(row => row.addEventListener("click", () => {
176
+ state.topic = state.topic === row.dataset.topic ? null : row.dataset.topic;
177
+ document.querySelectorAll(".topic-row").forEach(r => r.classList.toggle("active", r.dataset.topic === state.topic));
178
+ loadFiles();
179
+ }));
180
+ }
181
+ $("#clear-filters").addEventListener("click", () => {
182
+ state.kinds.clear(); state.topic = null; state.node = null; state.q = ""; $("#search").value = "";
183
+ document.querySelectorAll("#kind-filters input").forEach(cb => cb.checked = false);
184
+ document.querySelectorAll(".topic-row").forEach(r => r.classList.remove("active"));
185
+ loadFacets().then(loadFiles);
186
+ });
187
+ }
188
+
189
+ async function loadFiles() {
190
+ const p = new URLSearchParams();
191
+ if (state.kinds.size) p.set("kind", [...state.kinds].join(","));
192
+ if (state.topic) p.set("topic", state.topic);
193
+ if (state.node) p.set("node", state.node);
194
+ if (state.q) p.set("q", state.q);
195
+ state.files = await api("/api/files?" + p.toString());
196
+ $("#file-list").innerHTML = state.files.map(f =>
197
+ `<li data-id="${f.id}" class="${f.id === state.selected ? "selected" : ""}">
198
+ <div class="fname">${esc(f.name)}</div>
199
+ <div class="fmeta">${esc(f.kind)} · ${fmtSize(f.size_bytes)} · ${fmtDate(f.mtime)}</div>
200
+ <div>${f.tags.map(t => `<span class="tag">${esc(t)}</span>`).join("")}</div>
201
+ </li>`).join("") || `<li class="placeholder">No files match these filters.</li>`;
202
+ document.querySelectorAll("#file-list li[data-id]").forEach(li =>
203
+ li.addEventListener("click", () => selectFile(li.dataset.id)));
204
+ if (state.graph) drawGraph(); // the graph mirrors the list's filters
205
+ }
206
+
207
+ async function selectFile(id) {
208
+ state.selected = id;
209
+ document.querySelectorAll("#file-list li").forEach(li => li.classList.toggle("selected", li.dataset.id === id));
210
+ const d = await api("/api/file/" + encodeURIComponent(id));
211
+ const pane = $("#preview-pane");
212
+ let h = `<h2>${esc(d.name)}</h2>
213
+ <div class="meta">${esc(d.kind)} · ${fmtSize(d.size_bytes)} · modified ${fmtDate(d.mtime)}<br>${esc(d.path)}</div>
214
+ ${d.title ? `<div class="meta">Title: ${esc(d.title)}</div>` : ""}
215
+ ${d.author ? `<div class="meta">Author: ${esc(d.author)}</div>` : ""}
216
+ <div>${d.tags.map(t => `<span class="tag">${esc(t)}</span>`).join("")}</div>
217
+ ${d.summary ? `<p class="summary">${esc(d.summary)}</p>` : ""}
218
+ ${d.entities && d.entities.length ? `<div class="meta">Entities: ${d.entities.map(esc).join(", ")}</div>` : ""}`;
219
+ if (d.data_review) {
220
+ const r = d.data_review;
221
+ const summ = Object.fromEntries((r.summarize || []).map(s => [s.name, s]));
222
+ const hasSumm = (r.summarize || []).length > 0;
223
+ h += `<h2>Data review</h2>
224
+ <div class="meta">${r.row_count != null ? r.row_count + " rows · " : ""}${r.columns.length} columns</div>
225
+ <table><tr><th>column</th><th>type</th><th>nulls</th>${hasSumm ? "<th>min</th><th>max</th><th>unique≈</th>" : ""}</tr>${r.columns.map(c => { const s = summ[c.name] || {}; return `<tr><td>${esc(c.name)}</td><td>${esc(c.type || "")}</td><td>${r.nulls[c.name] ?? ""}</td>${hasSumm ? `<td>${esc(s.min ?? "")}</td><td>${esc(s.max ?? "")}</td><td>${esc(s.approx_unique ?? "")}</td>` : ""}</tr>`; }).join("")}</table>
226
+ <div><code id="sample-query">${esc(r.sample_query)}</code><button class="copy-btn" id="copy-query">Copy</button></div>`;
227
+ } else if (d.text_preview) {
228
+ h += `<h2>Preview</h2><pre>${esc(d.text_preview)}</pre>`;
229
+ }
230
+ if (d.relations && d.relations.length) {
231
+ const names = Object.fromEntries(state.files.map(f => [f.id, f.name]));
232
+ h += `<h2>Linked files</h2>` + d.relations.slice(0, 10).map(rel =>
233
+ `<div class="meta">${esc(rel.kind)} — ${esc(rel.detail)} — ${esc(names[rel.other_id] || rel.other_id)}</div>`).join("");
234
+ }
235
+ pane.innerHTML = h;
236
+ const copy = $("#copy-query");
237
+ if (copy) copy.addEventListener("click", () => navigator.clipboard.writeText($("#sample-query").textContent));
238
+ if (state.view === "graph") highlightSelected();
239
+ // Stacked (mobile) layout: the preview sits below the list, so
240
+ // bring it into view after a tap.
241
+ if (window.innerWidth <= 920) pane.scrollIntoView({ behavior: "smooth", block: "start" });
242
+ }
243
+
244
+ async function loadGraph() {
245
+ if (!state.graph) state.graph = await api("/api/graph");
246
+ const kinds = ["shared_topic", "shared_entity", "reference", "same_folder"];
247
+ $("#edge-toggles").innerHTML = kinds.map(k =>
248
+ `<label><input type="checkbox" data-edge="${k}" ${state.edgeKinds.has(k) ? "checked" : ""}> ${k.replace("_", " ")}</label>`).join("");
249
+ document.querySelectorAll("#edge-toggles input").forEach(cb => cb.addEventListener("change", () => {
250
+ cb.checked ? state.edgeKinds.add(cb.dataset.edge) : state.edgeKinds.delete(cb.dataset.edge);
251
+ drawGraph();
252
+ }));
253
+ drawGraph();
254
+ }
255
+
256
+ function highlightSelected() {
257
+ if (!state.chart || !state.nodeIndex) return;
258
+ const idx = state.nodeIndex[state.selected];
259
+ if (idx == null) return;
260
+ state.chart.dispatchAction({ type: "select", seriesIndex: 0, dataIndex: idx });
261
+ }
262
+
263
+ function drawGraph() {
264
+ const g = state.graph; if (!g) return;
265
+ if (typeof echarts === "undefined") {
266
+ $("#edge-info").textContent = "ECharts failed to load — graph unavailable.";
267
+ return;
268
+ }
269
+ if (!state.chart) {
270
+ state.chart = echarts.init($("#graph-chart"));
271
+ state.chart.on("click", (params) => {
272
+ if (params.dataType === "node") selectFile(params.data.id);
273
+ else if (params.dataType === "edge") {
274
+ const names = Object.fromEntries(state.graph.nodes.map(n => [n.id, n.name]));
275
+ $("#edge-info").textContent =
276
+ `${names[params.data.source]} — ${names[params.data.target]}: ${params.data.edgeKind} (${params.data.detail})`;
277
+ }
278
+ });
279
+ window.addEventListener("resize", () => state.chart && state.chart.resize());
280
+ }
281
+ // The graph shows exactly the files the list shows: topic, kind,
282
+ // topic-tree and search filters apply to both.
283
+ const visibleIds = new Set(state.files.map(f => f.id));
284
+ const visNodes = g.nodes.filter(n => visibleIds.has(n.id));
285
+ const visIds = new Set(visNodes.map(n => n.id));
286
+ const visEdges = g.edges.filter(e => visIds.has(e.a) && visIds.has(e.b));
287
+ const kindNames = [...new Set(visNodes.map(n => n.kind))];
288
+ const categories = kindNames.map(k => ({ name: k, itemStyle: { color: KIND_COLORS[k] || "#999" } }));
289
+ const degree = {};
290
+ visEdges.forEach(e => { degree[e.a] = (degree[e.a] || 0) + 1; degree[e.b] = (degree[e.b] || 0) + 1; });
291
+ const bigN = visNodes.length > 120;
292
+ state.nodeIndex = {};
293
+ const nodes = visNodes.map((n, i) => {
294
+ state.nodeIndex[n.id] = i;
295
+ return {
296
+ id: n.id, name: n.name.length > 24 ? n.name.slice(0, 23) + "…" : n.name,
297
+ fullName: n.name, kind: n.kind, tags: n.tags || [],
298
+ category: kindNames.indexOf(n.kind),
299
+ symbolSize: bigN ? Math.min(22, 8 + (degree[n.id] || 0) * 0.8)
300
+ : Math.min(34, 12 + (degree[n.id] || 0) * 1.6),
301
+ itemStyle: { color: KIND_COLORS[n.kind] || "#999" },
302
+ };
303
+ });
304
+ const links = visEdges.filter(e => state.edgeKinds.has(e.kind)).map(e => ({
305
+ source: e.a, target: e.b, edgeKind: e.kind, detail: e.detail,
306
+ lineStyle: { color: EDGE_COLORS[e.kind] || "#bbb", type: e.kind === "same_folder" ? "dashed" : "solid", opacity: 0.75 },
307
+ }));
308
+ const big = visNodes.length > 250;
309
+ state.chart.setOption({
310
+ tooltip: {
311
+ formatter: (p) => {
312
+ if (p.dataType === "node")
313
+ return `${esc(p.data.fullName)}<br/>${esc(p.data.kind)}` +
314
+ (p.data.tags.length ? `<br/>${p.data.tags.map(esc).join(", ")}` : "");
315
+ if (p.dataType === "edge") return `${esc(p.data.edgeKind)}: ${esc(p.data.detail)}`;
316
+ return "";
317
+ },
318
+ },
319
+ legend: [{ data: kindNames, top: 4, left: 8 }],
320
+ series: [{
321
+ type: "graph", layout: "force", roam: true, draggable: true,
322
+ categories, data: nodes, links,
323
+ selectedMode: "single",
324
+ select: { itemStyle: { borderColor: "#111", borderWidth: 3 } },
325
+ emphasis: { focus: "adjacency", lineStyle: { width: 3 } },
326
+ label: { show: !big, position: "bottom", fontSize: 11,
327
+ formatter: (p) => p.data.name },
328
+ labelLayout: { hideOverlap: true },
329
+ zoom: visNodes.length > 150 ? 0.55 : visNodes.length > 60 ? 0.8 : 1.05,
330
+ force: {
331
+ repulsion: visNodes.length > 300 ? 90 : 200,
332
+ edgeLength: [50, 150], gravity: 0.06, friction: 0.15,
333
+ layoutAnimation: true,
334
+ },
335
+ lineStyle: { curveness: 0.08 },
336
+ }],
337
+ }, true);
338
+ highlightSelected();
339
+ state.chart.resize();
340
+ }
341
+
342
+ function setView(v) {
343
+ state.view = v;
344
+ $("#btn-list").classList.toggle("active", v === "list");
345
+ $("#btn-graph").classList.toggle("active", v === "graph");
346
+ // Desktop shows the list and the graph side by side; the toggle only
347
+ // switches panes on narrow (mobile) layouts. Graph is the default.
348
+ const mobile = window.innerWidth <= 920;
349
+ $("#list-pane").classList.toggle("hidden", mobile && v !== "list");
350
+ $("#graph-pane").classList.toggle("hidden", mobile && v !== "graph");
351
+ if (!mobile || v === "graph") {
352
+ loadGraph();
353
+ if (state.chart) setTimeout(() => state.chart.resize(), 50);
354
+ }
355
+ }
356
+
357
+ // Download via fetch + Blob: works against the live server and inside
358
+ // the static demo snapshot, whose API stub answers the same URL (a
359
+ // plain anchor navigation would leave the demo site and 404).
360
+ $("#cypher-link").addEventListener("click", async (e) => {
361
+ e.preventDefault();
362
+ try {
363
+ const r = await fetch("/api/export.cypher");
364
+ const text = await r.text();
365
+ const a = document.createElement("a");
366
+ a.href = URL.createObjectURL(new Blob([text], { type: "text/plain" }));
367
+ a.download = "folderlake.cypher";
368
+ a.click();
369
+ setTimeout(() => URL.revokeObjectURL(a.href), 5000);
370
+ } catch (err) {
371
+ window.location = "/api/export.cypher";
372
+ }
373
+ });
374
+
375
+ $("#btn-list").addEventListener("click", () => setView("list"));
376
+ $("#btn-graph").addEventListener("click", () => setView("graph"));
377
+ let wasMobile = window.innerWidth <= 920;
378
+ window.addEventListener("resize", () => {
379
+ const m = window.innerWidth <= 920;
380
+ if (m !== wasMobile) { wasMobile = m; setView(state.view); }
381
+ });
382
+ let searchTimer;
383
+ $("#search").addEventListener("input", (e) => {
384
+ clearTimeout(searchTimer);
385
+ searchTimer = setTimeout(() => { state.q = e.target.value.trim(); loadFiles(); }, 250);
386
+ });
387
+
388
+ (async () => {
389
+ const meta = await api("/api/meta");
390
+ $("#folder-label").textContent = meta.folder + ` — ${meta.catalog_bytes ? (meta.catalog_bytes / 1024).toFixed(0) + " KB catalog" : ""}`;
391
+ await loadFacets();
392
+ await loadFiles();
393
+ setView(state.view);
394
+ })();
395
+ </script>
396
+ </body>
397
+ </html>
@@ -0,0 +1,278 @@
1
+ Metadata-Version: 2.4
2
+ Name: folderlake
3
+ Version: 0.2.0
4
+ Summary: Point it at a folder. Know what's in it. A local file catalog with topics, metadata and a relationship graph, stored in one DuckDB file.
5
+ Requires-Python: >=3.11
6
+ Description-Content-Type: text/markdown
7
+ License-File: LICENSE
8
+ Requires-Dist: duckdb
9
+ Requires-Dist: python-docx
10
+ Requires-Dist: pypdf
11
+ Requires-Dist: openpyxl
12
+ Provides-Extra: llm
13
+ Requires-Dist: litellm>=1.40; extra == "llm"
14
+ Provides-Extra: mcp
15
+ Requires-Dist: mcp>=1.0; extra == "mcp"
16
+ Dynamic: license-file
17
+
18
+ # FolderLake
19
+
20
+ **Point it at a folder. Know what's in it — an LLM reads the files,
21
+ and an MCP server lets your other agents query the result.**
22
+
23
+ FolderLake scans a folder once and builds a local catalog of every file
24
+ in it — **LLM topic tags, a one-line summary and key entities per
25
+ file**, plus a graph of how the files relate to each other. The
26
+ catalog is a single DuckDB file that lives next to your data. No
27
+ server, no account, nothing leaves your machine.
28
+
29
+ - **LLM topics** — plug in a model (Ollama local, OpenAI, Anthropic,
30
+ any OpenAI-compatible endpoint) for phrase tags and real summaries;
31
+ a built-in TF-IDF tagger covers folders with no model configured.
32
+ - **MCP built in** — `folderlake-mcp` exposes the catalog to Claude
33
+ Code, Claude Desktop and other MCP clients: search, browse, inspect
34
+ and traverse the graph without opening the UI.
35
+ - **Zero setup** — one `pip install`, one command, a web UI opens.
36
+ - **Fast** — 1,000 mixed files scan in ~13 s; rescans only touch new or
37
+ changed files (~0.1 s when nothing changed).
38
+ - **Light** — a 1,000-file catalog is ~1.6 MB on disk.
39
+ - **Many formats** — txt, md, html, docx, pdf, csv, tsv, json, parquet,
40
+ xlsx and sqlite out of the box.
41
+
42
+ ## Live demo
43
+
44
+ **[Open the demo](https://daizack.github.io/FolderLake/)** — a real
45
+ FolderLake catalog of 193 public files (AG News, 20 Newsgroups, Wikipedia
46
+ articles and the seaborn datasets), browsable in your browser.
47
+
48
+ The demo's topic tags and summaries were generated by **Muse** (Meta's
49
+ AI assistant) acting as FolderLake's LLM tier — see
50
+ [How the demo was made](#how-the-demo-was-made). With no LLM configured,
51
+ FolderLake falls back to a built-in TF-IDF tagger; the demo shows what
52
+ the LLM tier adds: phrase tags like `car air conditioning` instead of
53
+ single words, and a real one-sentence summary per file.
54
+
55
+ Two things worth trying in the demo: click a topic in the tree — the
56
+ list and the graph filter together — and open the Wikipedia article
57
+ on the Titanic: it links straight to the Titanic passenger datasets.
58
+ FolderLake adds a *reference* link whenever a text names a dataset's
59
+ subject (a capitalised mention like "Titanic" or "Penguins"), on top
60
+ of shared topics, shared entities and explicit filename references.
61
+
62
+ ![List and relationship graph on one page — the graph is the default view, list and graph side by side](docs/screenshots/topics-tree.png)
63
+
64
+ ![A topic selected in the tree filters the list and the graph together](docs/screenshots/graph.png)
65
+
66
+ ![File preview with data review](docs/screenshots/file-preview.png)
67
+
68
+ <img src="docs/screenshots/mobile.png" alt="FolderLake on a phone" width="320">
69
+
70
+ ## Quick start
71
+
72
+ ```bash
73
+ pip install "git+https://github.com/DaiZack/FolderLake"
74
+ # (not on PyPI yet — GitHub is the source; or `pip install -e .` from a clone)
75
+
76
+ folderlake app # opens the FolderLake home page
77
+ ```
78
+
79
+ ![The FolderLake home page: history, new scans and settings](docs/screenshots/home.png)
80
+
81
+ That's the whole setup. `app` opens a home page in your browser right
82
+ away — no dialog, no waiting. Paste or browse to a folder, hit
83
+ **Scan folder**, and watch the progress bar; when the scan finishes
84
+ the catalog opens. Every catalog you build is remembered on the home
85
+ page (stored in a small SQLite registry in `~/.folderlake/`): reopen
86
+ one in a click, **Refresh** it to pick up added/removed files, or
87
+ **Delete** it. Each history entry shows the settings it was scanned
88
+ with. Inside a catalog, a **← FolderLake Home** link brings you back.
89
+
90
+ The catalog viewer itself: filters and the topic tree on the left,
91
+ the file list and the relationship graph side by side in the middle
92
+ (on a phone they take turns, graph first), and a preview pane on the
93
+ right. Clicking a topic filters the list and the graph together.
94
+ Data files get a *data review* in the preview
95
+ pane — schema, null counts, min/max, approximate uniques — plus a
96
+ ready-to-run DuckDB query.
97
+
98
+ Prefer the terminal:
99
+
100
+ ```bash
101
+ folderlake scan ~/Documents/notes # build or update the catalog
102
+ folderlake serve ~/Documents/notes # rescan + open the viewer
103
+ folderlake cypher ~/Documents/notes # export the graph as Neo4j Cypher
104
+ ```
105
+
106
+ Every command is incremental: unchanged files are skipped by file stats
107
+ alone, so re-running on a big folder is cheap.
108
+
109
+ ## Using an LLM for topics
110
+
111
+ With no LLM configured you get rule-based TF-IDF phrase tags. Point
112
+ FolderLake at any chat model and each new or changed file gets LLM tags,
113
+ a one-sentence summary and key entities instead (results are cached by
114
+ content hash, so each file is sent exactly once). In the app, the
115
+ home page's Settings card holds the same options — no environment
116
+ variables needed; the environment variables below remain for the
117
+ CLI and scripts:
118
+
119
+ ```bash
120
+ pip install "folderlake[llm] @ git+https://github.com/DaiZack/FolderLake"
121
+ # (adds LiteLLM as the transport)
122
+
123
+ # Local model via Ollama (default provider)
124
+ FOLDERLAKE_LLM_MODEL=qwen3 folderlake scan ~/notes --llm
125
+
126
+ # Any OpenAI-compatible endpoint (OpenAI, DeepSeek, Groq, LM Studio, vLLM…)
127
+ FOLDERLAKE_LLM_PROVIDER=openai \
128
+ FOLDERLAKE_LLM_MODEL=gpt-4o-mini \
129
+ FOLDERLAKE_LLM_API_KEY=sk-… \
130
+ folderlake scan ~/notes --llm
131
+
132
+ # Anthropic
133
+ FOLDERLAKE_LLM_PROVIDER=anthropic \
134
+ FOLDERLAKE_LLM_MODEL=claude-haiku-4-5 \
135
+ FOLDERLAKE_LLM_API_KEY=sk-ant-… \
136
+ folderlake scan ~/notes --llm
137
+ ```
138
+
139
+ Data files are never sent raw: the LLM sees only a profile card
140
+ (`DESCRIBE` + `SUMMARIZE` output — column names and aggregate stats),
141
+ never sample row values. If the LLM is unreachable, the scan falls back
142
+ to rule topics and says so in its stats.
143
+
144
+ ## Let other agents query it (MCP)
145
+
146
+ FolderLake ships an MCP server with six read-only tools — `stats`,
147
+ `search`, `list_files`, `file_detail`, `related`, `topics` — so any MCP
148
+ client (Claude Code, Cursor, your own agent) can ask questions about a
149
+ catalog without opening files:
150
+
151
+ ```bash
152
+ pip install "folderlake[mcp] @ git+https://github.com/DaiZack/FolderLake"
153
+ folderlake mcp ~/Documents/notes # stdio MCP server
154
+ # or the folderlake-mcp entry point
155
+ ```
156
+
157
+ ## Tech stack
158
+
159
+ | Layer | Choice |
160
+ |---|---|
161
+ | Language | Python 3.11+, standard library first |
162
+ | Catalog store | **DuckDB** — one `.duckdb` file per scanned folder (files, topics, entities, relations, extracted text, topic tree) |
163
+ | Extraction | `python-docx`, `pypdf`, `openpyxl`, stdlib for txt/md/html/csv/json/sqlite; DuckDB itself reads parquet/csv |
164
+ | Rule topics | Corpus TF-IDF with phrase mining (English word bigrams, CJK bigrams/trigrams), title boost, boilerplate suppression |
165
+ | Topic tree | Agglomerative hierarchical clustering over TF-IDF, per file kind; labels by term lift; rendered as a collapsible tree |
166
+ | LLM tier | Provider layer over **LiteLLM** (optional extra) with a built-in client as fallback — Ollama, OpenAI-compatible, Anthropic, Hugging Face |
167
+ | Viewer | One HTML file, vanilla JS + **Apache ECharts** 5.5.1 (vendored, the only JS dependency), served by stdlib `http.server`; responsive down to phone widths |
168
+ | Graph export | Idempotent Neo4j Cypher (`MERGE` script) |
169
+ | Agent access | Official `mcp` Python SDK (optional extra), stdio transport |
170
+ | App state | SQLite registry (stdlib `sqlite3`) in `~/.folderlake/registry.sqlite3` — catalog history with per-scan settings snapshots, plus the user settings |
171
+ | Tests | pytest — 59 tests covering extractors, topics, relations, hierarchy, server and MCP |
172
+
173
+ ### The ML / NLP pipeline, briefly
174
+
175
+ - **Tokenization** — English unigrams + word bigrams; Chinese character
176
+ bigrams/trigrams; data files contribute joined column-name phrases
177
+ (`total_bill` → "total bill").
178
+ - **Phrase mining** — candidate tags are scored with TF-IDF (smoothed
179
+ IDF, title/filename boost ×3, phrase-length bonus); a phrase's
180
+ component words are suppressed so tags read like `car air
181
+ conditioning`, not `car`, `air`, `conditioning`.
182
+ - **LLM classification** — with a model configured, each file's
183
+ "hotspot" segments (~10k chars: opening, densest passages, closing)
184
+ go to the model, which returns structured JSON (tags / summary /
185
+ entities); results are cached by content hash. Data files are
186
+ described by schema + profile statistics only — sample values never
187
+ reach the topic model.
188
+ - **Topic tree** — per-kind agglomerative hierarchical clustering
189
+ (centroid linkage, cosine over L2-normalized TF-IDF vectors, corpus
190
+ document-frequency hygiene); node labels are picked by term *lift*
191
+ against the whole kind, and "forced marriage" nodes whose halves
192
+ barely resemble each other are flattened away.
193
+ - **Entities** — capitalized-sequence extraction (≥2 mentions) in the
194
+ rule tier; the model's entity list in the LLM tier.
195
+ - **Embeddings** — deliberately not used anywhere; see
196
+ [Limitations](#limitations-honestly).
197
+
198
+ ## Demo data
199
+
200
+ The 193 files in the live demo are public material, assembled to look
201
+ like a real mixed folder:
202
+
203
+ | Kind | Count | Source |
204
+ |---|---|---|
205
+ | Texts | 130 | Posts sampled from the **20 Newsgroups** corpus, plus **Wikipedia** article excerpts (English and Chinese), as `.txt` |
206
+ | Documents | 33 | **AG News** articles packaged as `.pdf` / `.docx` / `.html`, plus longer Wikipedia articles |
207
+ | Data | 30 | The 10 **seaborn** example datasets (tips, iris, penguins, titanic, diamonds, flights, mpg, planets, exercise, attention), each in 3 formats (CSV / JSON / Parquet or XLSX) |
208
+
209
+ Two Wikipedia leads — *RMS Titanic* and *Penguin* — are included on
210
+ purpose: they are what the Titanic and penguins datasets are *about*,
211
+ so the demo contains genuine text→data links.
212
+
213
+ Wikipedia text is CC BY-SA; 20 Newsgroups and AG News are standard
214
+ public research corpora; the seaborn datasets ship with the
215
+ BSD-licensed seaborn library. The corpus itself is not distributed
216
+ with this repository — only the rendered catalog snapshot is
217
+ published, and articles are trimmed to excerpts.
218
+
219
+ ## How the demo was made
220
+
221
+ The live demo is a static snapshot of a real catalog, produced like
222
+ this (all steps reproducible from this repo):
223
+
224
+ 1. A 193-file corpus of public datasets was scanned with
225
+ `folderlake scan --llm`.
226
+ 2. The "LLM" was **Muse**, served through the stub in
227
+ [`tools/llm_stub.py`](tools/llm_stub.py): it answers FolderLake's
228
+ ordinary OpenAI-compatible requests with responses Muse authored
229
+ for each file — so the demo exercises the real provider layer,
230
+ prompt packaging, JSON parsing and hash caching, with Muse as the
231
+ model. The scan stats reported `llm: 193, rule: 0`.
232
+ 3. The viewer was snapshotted to a single self-contained HTML file
233
+ (kept in [`demo/`](demo/) in this repo), which GitHub Actions
234
+ deploys to GitHub Pages on every push that touches it.
235
+
236
+ To recreate the sample folder from the screenshots in early docs:
237
+
238
+ ```bash
239
+ python tools/make_demo_folder.py # writes ./demo_folder (bike shop + garden)
240
+ ```
241
+
242
+ ## Benchmarks
243
+
244
+ Measured in a small shared-CPU sandbox VM (absolute numbers will be
245
+ better on a real machine; treat them as a floor):
246
+
247
+ | Operation | Result |
248
+ |---|---|
249
+ | Full scan, 1,000 mixed files (rule topics) | 12.6 s, catalog 1.59 MB |
250
+ | Incremental rescan, nothing changed | 0.09 s |
251
+ | Full scan, 2,000 files (topic pipeline v2) | ~38 s |
252
+ | Topic tree build, 193 files | ~1.5 s |
253
+
254
+ ## Limitations, honestly
255
+
256
+ - Relations grow quadratically before deduplication — very large
257
+ folders (5,000+ files) get heavy graph tables; per-tag edge budgets
258
+ are planned but not yet shipped.
259
+ - The topic tree is built for kinds with at most 400 files (the
260
+ clustering is O(N²)); larger kinds keep flat topic facets.
261
+ - Search uses a `LIKE` fallback when DuckDB's FTS extension is
262
+ unavailable.
263
+ - Embeddings are deliberately **not** a dependency: if they are added,
264
+ they will arrive through the same provider layer as the LLM tier,
265
+ never as a bundled torch install.
266
+
267
+ ## Development
268
+
269
+ ```bash
270
+ pip install -e ".[llm,mcp]"
271
+ pytest # 59 tests
272
+ ```
273
+
274
+ Demo data: see [Demo data](#demo-data) above.
275
+
276
+ ## License
277
+
278
+ MIT — see [LICENSE](LICENSE).