pikuri-fileindex 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/README.md +114 -0
- data/lib/pikuri/fileindex/extension.rb +36 -0
- data/lib/pikuri/fileindex/local_search.rb +402 -0
- data/lib/pikuri/fileindex/query.rb +50 -0
- data/lib/pikuri/fileindex/read.rb +231 -0
- data/lib/pikuri/fileindex/recoll.rb +372 -0
- data/lib/pikuri/fileindex/roots.rb +109 -0
- data/lib/pikuri/fileindex/search.rb +288 -0
- data/lib/pikuri-fileindex.rb +70 -0
- metadata +75 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 4d05f84868f8fc4907f3a16f3ec7e5c8a4bd6b57f476b49ad4abe0bb63d0def4
|
|
4
|
+
data.tar.gz: af99d134e391f4ba46a76ec48c6650b1afb6d26e00a3b93157b404262ef0b832
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 7d56e04cd9c67708fd2c366fa833c64424eea69dacfa84909e3b1f4e610de9cfa4e2faef55ab8ae97ee6aa059324f14695c902f6961636caf41f001b1a455553
|
|
7
|
+
data.tar.gz: 1931986bd6fe2ebcb2cd552994f720236d1b582b45022a3a35dea2d2d9e935444a15349d396a1c13e3f0b38d2e742a258d193eb4898217cb25a182a6c1e20726
|
data/README.md
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
# pikuri-fileindex
|
|
2
|
+
|
|
3
|
+
Search your files through the index your desktop already keeps, for the
|
|
4
|
+
[pikuri](https://github.com/mvysny/pikuri) AI-assistant toolkit. Ask
|
|
5
|
+
"find the Kettle & Co. invoice" and the agent gets back the PDF's path
|
|
6
|
+
and a 40-word snippet with the amount in it, without pikuri ever opening
|
|
7
|
+
the PDF itself.
|
|
8
|
+
|
|
9
|
+
Provides:
|
|
10
|
+
- `fileindex_search` (`Pikuri::Fileindex::Search`): keyword search, one
|
|
11
|
+
snippet per file (with localsearch, a file of up to 600 characters comes
|
|
12
|
+
whole), optionally limited to one folder. A file matches when
|
|
13
|
+
it holds any of the words (`rent lease vuokra`), or all the words in
|
|
14
|
+
double quotes, in any order; files with every word come first. It says
|
|
15
|
+
how many files match in all ("Showing 10 of 37"), so "all shown" means
|
|
16
|
+
there is no other.
|
|
17
|
+
- `fileindex_read` (`Pikuri::Fileindex::Read`): a file's full text as the
|
|
18
|
+
index extracted it, paged like `read`, with an optional `pattern` that
|
|
19
|
+
returns only the matching lines.
|
|
20
|
+
- `Pikuri::Fileindex::Roots`: the folders you let the agent reach. The
|
|
21
|
+
agent can't search or read anything outside them, or any hidden file
|
|
22
|
+
inside them.
|
|
23
|
+
- Two backends: GNOME's **localsearch** (queried through `tinysparql`)
|
|
24
|
+
and **recoll** (through `recollq`). `Pikuri::Fileindex.pick` takes the
|
|
25
|
+
first one that is ready.
|
|
26
|
+
|
|
27
|
+
## Why borrow the desktop's index
|
|
28
|
+
|
|
29
|
+
- **No GPU bill, no daemon of ours.** GNOME already indexes your files
|
|
30
|
+
and keeps the index fresh as you save. Building our own embedding
|
|
31
|
+
index of `~/Documents` kept a laptop GPU at 100% for hours.
|
|
32
|
+
- **The desktop parses the hostile files, not pikuri.** localsearch
|
|
33
|
+
extracts the text of a `.docx` or a PDF in its own sandboxed process.
|
|
34
|
+
A malicious file in Downloads is never parsed by pikuri.
|
|
35
|
+
- **Office documents come free**: the index already holds the text of
|
|
36
|
+
`.docx`, `.odt` and spreadsheets (on Ubuntu, once
|
|
37
|
+
`localsearch-extractor-office` is installed).
|
|
38
|
+
- **Keyword search can say "that's every match"**, which a vector search
|
|
39
|
+
can't.
|
|
40
|
+
|
|
41
|
+
It never builds, configures or refreshes an index; it only reads one.
|
|
42
|
+
|
|
43
|
+
## Requirements
|
|
44
|
+
|
|
45
|
+
One of:
|
|
46
|
+
- **localsearch 3.8 or newer** with the `tinysparql` CLI (Ubuntu 26.04
|
|
47
|
+
LTS and newer; Debian/Ubuntu's `localsearch` package pulls in
|
|
48
|
+
`tinysparql`). Older Tracker 3 (`tracker3`) is not supported. On Ubuntu,
|
|
49
|
+
also install `localsearch-extractor-office`: without it, `.docx`, `.odt`,
|
|
50
|
+
`.xlsx` and epub files are found by name only, never by their text. Files
|
|
51
|
+
indexed before you install it stay that way, even after a restart
|
|
52
|
+
([a localsearch bug](https://gitlab.gnome.org/GNOME/localsearch/-/work_items/476)),
|
|
53
|
+
so run `localsearch reset` once afterwards: it erases the index and
|
|
54
|
+
localsearch builds it again, office text included. That takes a while;
|
|
55
|
+
`localsearch status` says when it's done.
|
|
56
|
+
- **recoll** with an index you built (`recollindex`). Run
|
|
57
|
+
`recollindex -m` to keep it fresh as files change.
|
|
58
|
+
|
|
59
|
+
What the index doesn't cover, the agent can't find: localsearch skips
|
|
60
|
+
hidden folders, anything inside a git checkout, and the text of source
|
|
61
|
+
code and `.csv` files; recoll covers only the folders in its `topdirs`.
|
|
62
|
+
|
|
63
|
+
## Install
|
|
64
|
+
|
|
65
|
+
```ruby
|
|
66
|
+
# Gemfile
|
|
67
|
+
gem 'pikuri-fileindex'
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
## Usage
|
|
71
|
+
|
|
72
|
+
```ruby
|
|
73
|
+
require 'pikuri-core'
|
|
74
|
+
require 'pikuri-fileindex'
|
|
75
|
+
|
|
76
|
+
roots = Pikuri::Fileindex::Roots.new(dirs: %w[~/Desktop ~/Documents], private: true)
|
|
77
|
+
backend = Pikuri::Fileindex.pick(roots) # raises Pikuri::Fileindex::Unavailable, saying why
|
|
78
|
+
agent = Pikuri::Agent.new(transport:, system_prompt:) do |c|
|
|
79
|
+
c.add_extension Pikuri::Fileindex::Extension.new(roots:, backend:)
|
|
80
|
+
end
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
**Name your folders.** A stock GNOME install indexes your whole home
|
|
84
|
+
folder: `~/work`, `~/snap`, your backups. Only the roots keep an agent
|
|
85
|
+
you gave `~/Documents` out of the rest.
|
|
86
|
+
|
|
87
|
+
`private:` says the folders hold data worth stealing; `trusted:` says
|
|
88
|
+
every file in them is your own, so nothing a stranger wrote (not
|
|
89
|
+
Downloads, not a folder an agent that reads your mail writes into).
|
|
90
|
+
Both default to `false`, and they set the tools' trifecta legs.
|
|
91
|
+
|
|
92
|
+
Pick once per process and hand the backend to every agent: checking
|
|
93
|
+
readiness runs one query per root. When neither index holds a file
|
|
94
|
+
under the roots, the `Unavailable` message names what is missing for
|
|
95
|
+
each index and how to fix it; show it to the user. An empty Desktop
|
|
96
|
+
among other roots is fine, and only logs a warning.
|
|
97
|
+
|
|
98
|
+
## Try it
|
|
99
|
+
|
|
100
|
+
`bin/pikuri-briefing` prepares you for a customer meeting from your own
|
|
101
|
+
hand-written notes: the timeline, what was discussed last, what was
|
|
102
|
+
promised. The model runs locally and the agent has no internet access.
|
|
103
|
+
|
|
104
|
+
```
|
|
105
|
+
./pikuri-fileindex/bin/pikuri-briefing --source ~/notes "brief me on Acme"
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
`bin/pikuri-secretary`, at the repository root, searches your Desktop
|
|
109
|
+
and Documents the same way.
|
|
110
|
+
|
|
111
|
+
## Read more
|
|
112
|
+
|
|
113
|
+
Design notes and the backend survey:
|
|
114
|
+
[`design/architecture.md`](design/architecture.md).
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pikuri
|
|
4
|
+
module Fileindex
|
|
5
|
+
# Wires +fileindex_search+ and +fileindex_read+ over one backend and one
|
|
6
|
+
# set of roots, so the two tools can never disagree about which folders
|
|
7
|
+
# the agent reaches:
|
|
8
|
+
#
|
|
9
|
+
# roots = Pikuri::Fileindex::Roots.new(dirs: %w[~/Desktop ~/Documents], private: true)
|
|
10
|
+
# backend = Pikuri::Fileindex.pick(roots) # once per process; raises Unavailable
|
|
11
|
+
# Pikuri::Agent.new(transport:, system_prompt:) do |c|
|
|
12
|
+
# c.add_extension Pikuri::Fileindex::Extension.new(roots:, backend:)
|
|
13
|
+
# end
|
|
14
|
+
#
|
|
15
|
+
# The backend comes in rather than being picked here: picking probes the
|
|
16
|
+
# index once per root, and an extension is built anew for every agent.
|
|
17
|
+
class Extension
|
|
18
|
+
include Pikuri::Agent::Extension
|
|
19
|
+
|
|
20
|
+
# @param roots [Roots] the folders both tools reach
|
|
21
|
+
# @param backend [Module] a ready backend, from {Fileindex.pick}
|
|
22
|
+
def initialize(roots:, backend:)
|
|
23
|
+
@roots = roots
|
|
24
|
+
@backend = backend
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
# @param c [Pikuri::Agent::Configurator]
|
|
28
|
+
# @return [void]
|
|
29
|
+
def configure(c)
|
|
30
|
+
c.add_tool Search.new(backend: @backend, roots: @roots)
|
|
31
|
+
c.add_tool Read.new(backend: @backend, roots: @roots)
|
|
32
|
+
nil
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
end
|
|
@@ -0,0 +1,402 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pikuri
|
|
4
|
+
module Fileindex
|
|
5
|
+
# The GNOME desktop file index (+localsearch+, formerly tracker-miners),
|
|
6
|
+
# queried in SPARQL through the +tinysparql+ CLI. The stateless backend
|
|
7
|
+
# behind +fileindex_search+ / +fileindex_read+:
|
|
8
|
+
#
|
|
9
|
+
# LocalSearch.search(query: Query.parse('lease rent'), limit: 10, folders: ['/home/u/Documents'])
|
|
10
|
+
# # => [{ path: '/home/u/Documents/Housing/lease-agreement.pdf',
|
|
11
|
+
# # snippet: 'Residential >>Lease<< Agreement Landlord: …',
|
|
12
|
+
# # text_size: 4211, text: nil },
|
|
13
|
+
# # { path: '/home/u/Documents/Notes/flat.md',
|
|
14
|
+
# # snippet: '# Flat The >>rent<< is EUR 1,150 …',
|
|
15
|
+
# # text_size: 212, text: "# Flat\nThe rent is EUR 1,150, due on the 5th.\n…" }, …]
|
|
16
|
+
# LocalSearch.read_text('/home/u/Documents/Housing/lease-agreement.pdf')
|
|
17
|
+
# # => "Residential Lease Agreement\nLandlord: …" (nil when not indexed)
|
|
18
|
+
#
|
|
19
|
+
# A file matches when it holds every word of any one of the query's
|
|
20
|
+
# groups, by its name or its extracted text. Hits come one per file,
|
|
21
|
+
# those matching more groups first, then by relevance, each with one
|
|
22
|
+
# snippet of about {SNIPPET_WORDS} words around a match, matched words
|
|
23
|
+
# marked +>>like this<<+, and a short file's whole text too (up to
|
|
24
|
+
# {WHOLE_TEXT_CHARS}). PDFs and office docs come for free: the index
|
|
25
|
+
# extracted their text, so {.read_text} needs no extractor here.
|
|
26
|
+
#
|
|
27
|
+
# Needs +localsearch+ (the indexing daemon, a GNOME default) and the
|
|
28
|
+
# +tinysparql+ CLI, which Debian/Ubuntu's +localsearch+ package depends on.
|
|
29
|
+
#
|
|
30
|
+
# == Implementation details
|
|
31
|
+
#
|
|
32
|
+
# Why SPARQL and not +localsearch search+: that CLI sorts its hits by
|
|
33
|
+
# path, not relevance, before applying the limit, and cannot filter by
|
|
34
|
+
# folder — so a common word returned the first ten paths A–Z under +$HOME+.
|
|
35
|
+
#
|
|
36
|
+
# +tinysparql query+ prints a row as its cells joined by +", "+, which
|
|
37
|
+
# snippets and paths also contain. So every cell leaves SPARQL as
|
|
38
|
+
# +"v:" + ENCODE_FOR_URI(value)+: no comma, space or newline survives
|
|
39
|
+
# encoding, and the +v:+ marker sets data rows apart from the CLI's
|
|
40
|
+
# +Results:+ / +None+ lines, which are translated by locale.
|
|
41
|
+
#
|
|
42
|
+
# Index URLs use GLib's escaping ({.file_uri}); a folder filter or a read
|
|
43
|
+
# built with any other escaping silently matches nothing.
|
|
44
|
+
#
|
|
45
|
+
# Each group and each folder is its own bound parameter (+~group0+,
|
|
46
|
+
# +~folder0+, …); only those generated names are spliced into the query
|
|
47
|
+
# text, never a word or a path. A parameter's type is tinysparql's
|
|
48
|
+
# one-letter code (+s+, +i+): +string+ and +xsd:string+ are refused.
|
|
49
|
+
#
|
|
50
|
+
# Any-group matching is a +UNION+ of one +fts:match+ per group, because
|
|
51
|
+
# +fts:match+ has no OR: it reads +OR+, +NOT+ and parentheses as words.
|
|
52
|
+
module LocalSearch
|
|
53
|
+
# @return [String] the SPARQL query CLI.
|
|
54
|
+
BINARY = 'tinysparql'
|
|
55
|
+
|
|
56
|
+
# @return [String] the D-Bus name the localsearch daemon serves its
|
|
57
|
+
# index under; querying it starts the daemon through D-Bus activation.
|
|
58
|
+
DBUS_SERVICE = 'org.freedesktop.LocalSearch3'
|
|
59
|
+
|
|
60
|
+
# Not raised to show more: once a file's whole text fits the window,
|
|
61
|
+
# +fts:snippet+ may answer its title alone (measured: a 46-word
|
|
62
|
+
# invoice matching "invoice" gave +Kettle & Co. — >>Invoice<< #4471+
|
|
63
|
+
# at 55 words). {WHOLE_TEXT_CHARS} shows short files instead.
|
|
64
|
+
#
|
|
65
|
+
# @return [Integer] words of context in a snippet: enough to tell an
|
|
66
|
+
# invoice from a book that mentions one without reading either.
|
|
67
|
+
SNIPPET_WORDS = 40
|
|
68
|
+
|
|
69
|
+
# @return [Integer] longest stored text, in characters, that a hit
|
|
70
|
+
# carries whole: a few sentences, about the 500 characters a vector
|
|
71
|
+
# search's chunk shows. A snippet of a file this short leaves the
|
|
72
|
+
# rest, often the answer (an invoice's total), to a read call: on a
|
|
73
|
+
# 212-file test home (2026-11, 90 lookups, every answer in a short
|
|
74
|
+
# file), showing it whole cut the reads after a search from 73 to 10,
|
|
75
|
+
# for 13% more text a search.
|
|
76
|
+
WHOLE_TEXT_CHARS = 600
|
|
77
|
+
|
|
78
|
+
# @return [Array(String, String)] what wraps each matched word in a
|
|
79
|
+
# snippet, so the model sees which words hit.
|
|
80
|
+
MATCH_MARKERS = ['>>', '<<'].freeze
|
|
81
|
+
|
|
82
|
+
# @return [Logger]
|
|
83
|
+
LOGGER = Pikuri.logger_for('Fileindex::LocalSearch')
|
|
84
|
+
|
|
85
|
+
# The ranked, folder-scoped search: +%<match>s+ is {RANKED_MATCH} once
|
|
86
|
+
# per group, joined by +UNION+, and +%<under>s+ the folder filter,
|
|
87
|
+
# {.under}'s. +MAX+ over a file's rows keeps the better rank and the
|
|
88
|
+
# non-empty snippet. The +OPTIONAL+ measures the file's stored text
|
|
89
|
+
# and keeps it if short, whichever branch matched; a file indexed by
|
|
90
|
+
# name only leaves both unbound.
|
|
91
|
+
#
|
|
92
|
+
# @return [String]
|
|
93
|
+
SEARCH_QUERY = <<~SPARQL
|
|
94
|
+
SELECT (CONCAT("v:", ENCODE_FOR_URI(?url)) AS ?u)
|
|
95
|
+
(CONCAT("v:", ENCODE_FOR_URI(MAX(?snippet))) AS ?s)
|
|
96
|
+
(CONCAT("v:", STR(MAX(?length))) AS ?n)
|
|
97
|
+
(CONCAT("v:", ENCODE_FOR_URI(MAX(?short))) AS ?t)
|
|
98
|
+
WHERE {
|
|
99
|
+
%<match>s
|
|
100
|
+
?f nie:url ?url .
|
|
101
|
+
OPTIONAL {
|
|
102
|
+
?stored nie:isStoredAs ?f ;
|
|
103
|
+
nie:plainTextContent ?text .
|
|
104
|
+
BIND (STRLEN(?text) AS ?length)
|
|
105
|
+
BIND (IF(?length <= #{WHOLE_TEXT_CHARS}, ?text, "") AS ?short)
|
|
106
|
+
}
|
|
107
|
+
FILTER (tracker:available(nie:dataSource(?f)))
|
|
108
|
+
FILTER (%<under>s)
|
|
109
|
+
}
|
|
110
|
+
GROUP BY ?url
|
|
111
|
+
ORDER BY DESC(COUNT(DISTINCT ?group)) DESC(MAX(?rank)) ?url
|
|
112
|
+
LIMIT ~limit
|
|
113
|
+
SPARQL
|
|
114
|
+
|
|
115
|
+
# One group's branches in {SEARCH_QUERY}: the file matches by name
|
|
116
|
+
# (first branch, no snippet) or by extracted text (second). +%<group>s+
|
|
117
|
+
# is the group's parameter name, +%<index>d+ its number, which ranks a
|
|
118
|
+
# file by how many groups it matched.
|
|
119
|
+
#
|
|
120
|
+
# @return [String]
|
|
121
|
+
RANKED_MATCH = <<~SPARQL.chomp
|
|
122
|
+
{
|
|
123
|
+
?f fts:match ~%<group>s .
|
|
124
|
+
BIND (fts:rank(?f) AS ?rank)
|
|
125
|
+
BIND ("" AS ?snippet)
|
|
126
|
+
BIND (%<index>d AS ?group)
|
|
127
|
+
} UNION {
|
|
128
|
+
?elem fts:match ~%<group>s ;
|
|
129
|
+
nie:isStoredAs ?f .
|
|
130
|
+
BIND (fts:rank(?elem) AS ?rank)
|
|
131
|
+
BIND (fts:snippet(?elem, "#{MATCH_MARKERS[0]}", "#{MATCH_MARKERS[1]}", "…", #{SNIPPET_WORDS}) AS ?snippet)
|
|
132
|
+
BIND (%<index>d AS ?group)
|
|
133
|
+
}
|
|
134
|
+
SPARQL
|
|
135
|
+
|
|
136
|
+
# Every file {SEARCH_QUERY} matches, unranked and uncapped: the same
|
|
137
|
+
# branches ({MATCH}) and filters, no rank or snippet to compute.
|
|
138
|
+
#
|
|
139
|
+
# @return [String]
|
|
140
|
+
PATHS_QUERY = <<~SPARQL
|
|
141
|
+
SELECT DISTINCT (CONCAT("v:", ENCODE_FOR_URI(?url)) AS ?u)
|
|
142
|
+
WHERE {
|
|
143
|
+
%<match>s
|
|
144
|
+
?f nie:url ?url .
|
|
145
|
+
FILTER (tracker:available(nie:dataSource(?f)))
|
|
146
|
+
FILTER (%<under>s)
|
|
147
|
+
}
|
|
148
|
+
SPARQL
|
|
149
|
+
|
|
150
|
+
# One group's branches in {PATHS_QUERY}.
|
|
151
|
+
#
|
|
152
|
+
# @return [String]
|
|
153
|
+
MATCH = '{ ?f fts:match ~%<group>s . } UNION { ?elem fts:match ~%<group>s ; nie:isStoredAs ?f . }'
|
|
154
|
+
|
|
155
|
+
# @return [String] the stored text of the file at +~url+.
|
|
156
|
+
READ_QUERY = <<~SPARQL
|
|
157
|
+
SELECT (CONCAT("v:", ENCODE_FOR_URI(?text)) AS ?t)
|
|
158
|
+
WHERE {
|
|
159
|
+
?f nie:url ~url .
|
|
160
|
+
?elem nie:isStoredAs ?f ;
|
|
161
|
+
nie:plainTextContent ?text .
|
|
162
|
+
}
|
|
163
|
+
LIMIT 1
|
|
164
|
+
SPARQL
|
|
165
|
+
|
|
166
|
+
# Any one searchable file under +~folder0+: a folder alone gives a
|
|
167
|
+
# search nothing to find, and a file on an unmounted drive is filtered
|
|
168
|
+
# as {SEARCH_QUERY} filters it.
|
|
169
|
+
#
|
|
170
|
+
# @return [String]
|
|
171
|
+
PROBE_QUERY = <<~SPARQL
|
|
172
|
+
SELECT (CONCAT("v:", ENCODE_FOR_URI(?url)) AS ?u)
|
|
173
|
+
WHERE {
|
|
174
|
+
?f nie:url ?url .
|
|
175
|
+
?elem nie:isStoredAs ?f .
|
|
176
|
+
FILTER NOT EXISTS { ?elem a nfo:Folder }
|
|
177
|
+
FILTER (tracker:available(nie:dataSource(?f)))
|
|
178
|
+
FILTER (STRSTARTS(?url, ~folder0))
|
|
179
|
+
}
|
|
180
|
+
LIMIT 1
|
|
181
|
+
SPARQL
|
|
182
|
+
|
|
183
|
+
# @return [Regexp] a byte GLib leaves unescaped in a +file://+ URI
|
|
184
|
+
# (+g_filename_to_uri+'s set, measured against the index); anything
|
|
185
|
+
# else, non-ASCII included, is +%XX+ per byte.
|
|
186
|
+
URI_SAFE = %r{[A-Za-z0-9\-._~!$&'()*+,=:@/]}
|
|
187
|
+
|
|
188
|
+
# Raised when a tinysparql invocation exits non-zero — a recoverable
|
|
189
|
+
# failure the calling tool renders as +"Error: ..."+.
|
|
190
|
+
class CommandError < StandardError; end
|
|
191
|
+
|
|
192
|
+
# @return [String] short human name for this backend, used by the
|
|
193
|
+
# fileindex tools when prefixing an +"Error: ..."+ observation.
|
|
194
|
+
def self.label = 'localsearch'
|
|
195
|
+
|
|
196
|
+
# Uniform accessor mirroring {Recoll.limitations}; a constant here
|
|
197
|
+
# because this index's blind spots are the same on every host.
|
|
198
|
+
#
|
|
199
|
+
# @return [String]
|
|
200
|
+
def self.limitations = LIMITATIONS
|
|
201
|
+
|
|
202
|
+
# What this index will not surface, appended to the fileindex tools'
|
|
203
|
+
# LLM description so the model knows when to search the files directly
|
|
204
|
+
# instead. Measured; see +pikuri-fileindex/design/architecture.md+.
|
|
205
|
+
#
|
|
206
|
+
# @return [String]
|
|
207
|
+
LIMITATIONS = <<~LIMITS.chomp
|
|
208
|
+
Not covered by this index (fall back to a text/regex search over the files for these):
|
|
209
|
+
- Source code and other developer files are not content-indexed — only their names are. Search their text directly.
|
|
210
|
+
- Nothing inside a git checkout is indexed (any folder containing a .git). Search those files directly.
|
|
211
|
+
LIMITS
|
|
212
|
+
|
|
213
|
+
# Ranked full-text search under any of +folders+.
|
|
214
|
+
#
|
|
215
|
+
# @param query [Query] a file matches every word, stemmed, of any group
|
|
216
|
+
# @param limit [Integer] max files returned
|
|
217
|
+
# @param folders [Array<String>] absolute real paths of the folders to
|
|
218
|
+
# search under, subfolders included; at least one
|
|
219
|
+
# @return [Array<Hash>] +[{ path: String, snippet: String|nil,
|
|
220
|
+
# text_size: Integer|nil, text: String|nil }, …]+, best first, one
|
|
221
|
+
# per file; +text_size+ is the stored text's length in characters
|
|
222
|
+
# (SPARQL has no byte length; the two differ only by the non-ASCII
|
|
223
|
+
# share), +nil+ for a file indexed by name only; +text+ is the
|
|
224
|
+
# whole stored text when +text_size+ is at most {WHOLE_TEXT_CHARS}
|
|
225
|
+
# @raise [CommandError] if tinysparql exits non-zero
|
|
226
|
+
def self.search(query:, limit:, folders:)
|
|
227
|
+
raw = run(format(SEARCH_QUERY, match: union(RANKED_MATCH, query), under: under(folders.size)),
|
|
228
|
+
limit: limit, **group_args(query), **folder_args(folders))
|
|
229
|
+
parse_rows(raw).map do |url, snippet, length, text|
|
|
230
|
+
snippet = snippet&.gsub(/\s+/, ' ')&.strip
|
|
231
|
+
{ path: path_of(url), snippet: (snippet unless snippet.to_s.empty?), text_size: length&.to_i,
|
|
232
|
+
text: (text unless text.to_s.strip.empty?) }
|
|
233
|
+
end
|
|
234
|
+
end
|
|
235
|
+
|
|
236
|
+
# The path of every file {.search} would match, unranked and uncapped,
|
|
237
|
+
# so a caller can tell how many matched beyond its +limit+.
|
|
238
|
+
#
|
|
239
|
+
# @param query [Query]
|
|
240
|
+
# @param folders [Array<String>] as {.search}'s
|
|
241
|
+
# @return [Array<String>] absolute paths
|
|
242
|
+
# @raise [CommandError] if tinysparql exits non-zero
|
|
243
|
+
def self.paths(query:, folders:)
|
|
244
|
+
raw = run(format(PATHS_QUERY, match: union(MATCH, query), under: under(folders.size)),
|
|
245
|
+
**group_args(query), **folder_args(folders))
|
|
246
|
+
parse_rows(raw).map { |(url)| path_of(url) }
|
|
247
|
+
end
|
|
248
|
+
|
|
249
|
+
# Full stored plain text of an indexed file. Works for PDFs/office docs
|
|
250
|
+
# (extracted at index time).
|
|
251
|
+
#
|
|
252
|
+
# @param path [String] absolute path
|
|
253
|
+
# @return [String, nil] +nil+ when the path isn't indexed, has no stored
|
|
254
|
+
# text, or the query fails — the caller turns that into an LLM-facing
|
|
255
|
+
# message
|
|
256
|
+
def self.read_text(path)
|
|
257
|
+
row = parse_rows(run(READ_QUERY, url: file_uri(File.expand_path(path)))).first
|
|
258
|
+
text = row&.first
|
|
259
|
+
text && !text.strip.empty? ? text : nil
|
|
260
|
+
rescue CommandError
|
|
261
|
+
nil
|
|
262
|
+
end
|
|
263
|
+
|
|
264
|
+
# Why this index can't answer a search under +folders+ right now, for
|
|
265
|
+
# a human to act on; +nil+ when it can. An index that answers but holds
|
|
266
|
+
# no file under any of them is not ready either: every search would
|
|
267
|
+
# come back empty. Some empty folders are normal (an empty Desktop), so
|
|
268
|
+
# they only log a warning naming each.
|
|
269
|
+
#
|
|
270
|
+
# LocalSearch.unavailable_reason(folders: ['/home/u/notes'])
|
|
271
|
+
# # => "nothing under /home/u/notes is indexed. …" (nil when ready)
|
|
272
|
+
#
|
|
273
|
+
# Starts the localsearch daemon if it is installed but not running, as
|
|
274
|
+
# any query does (D-Bus activation).
|
|
275
|
+
#
|
|
276
|
+
# @param folders [Array<String>] absolute real paths
|
|
277
|
+
# @return [String, nil]
|
|
278
|
+
def self.unavailable_reason(folders:)
|
|
279
|
+
empty = folders.reject { |folder| parse_rows(run(PROBE_QUERY, folder0: prefix(folder))).any? }
|
|
280
|
+
if empty.size < folders.size
|
|
281
|
+
empty.each { |folder| LOGGER.warn("nothing under #{folder} is indexed; searches there find nothing") }
|
|
282
|
+
return nil
|
|
283
|
+
end
|
|
284
|
+
|
|
285
|
+
"nothing under #{folders.join(', ')} is indexed. Right after a first login GNOME may still be " \
|
|
286
|
+
'indexing: `localsearch status` shows how far it got. Otherwise add the folders under ' \
|
|
287
|
+
'GNOME Settings → Search. A folder inside a git checkout is never indexed.'
|
|
288
|
+
rescue Errno::ENOENT
|
|
289
|
+
"the '#{BINARY}' command is missing. Install the 'localsearch' and '#{BINARY}' " \
|
|
290
|
+
"packages: GNOME's desktop file index and its query tool."
|
|
291
|
+
rescue CommandError => e
|
|
292
|
+
"the index doesn't answer (#{e.message}). Is the 'localsearch' package " \
|
|
293
|
+
"installed? It is GNOME's desktop file indexer."
|
|
294
|
+
end
|
|
295
|
+
|
|
296
|
+
# The +file://+ URI the index stores for +path+, escaped the way GLib
|
|
297
|
+
# escapes it.
|
|
298
|
+
#
|
|
299
|
+
# LocalSearch.file_uri('/home/u/My Notes/ü#1.txt')
|
|
300
|
+
# # => "file:///home/u/My%20Notes/%C3%BC%231.txt"
|
|
301
|
+
#
|
|
302
|
+
# @param path [String] absolute path
|
|
303
|
+
# @return [String]
|
|
304
|
+
def self.file_uri(path)
|
|
305
|
+
escaped = path.b.gsub(/./mn) { |c| c.match?(URI_SAFE) ? c : format('%%%02X', c.ord) }
|
|
306
|
+
"file://#{escaped}"
|
|
307
|
+
end
|
|
308
|
+
|
|
309
|
+
# Parse +tinysparql query+ output into rows of decoded cells. Only rows
|
|
310
|
+
# whose every cell carries the +v:+ marker count (see the class doc).
|
|
311
|
+
#
|
|
312
|
+
# @param raw [String] the CLI's stdout
|
|
313
|
+
# @return [Array<Array<String, nil>>] one array of cells per row; an
|
|
314
|
+
# unbound cell (+(null)+) is +nil+
|
|
315
|
+
def self.parse_rows(raw)
|
|
316
|
+
raw.each_line(chomp: true).filter_map do |line|
|
|
317
|
+
next unless line.start_with?(' ')
|
|
318
|
+
|
|
319
|
+
cells = line.strip.split(', ')
|
|
320
|
+
next unless cells.all? { |cell| cell == '(null)' || cell.start_with?('v:') }
|
|
321
|
+
|
|
322
|
+
cells.map { |cell| cell == '(null)' ? nil : percent_decode(cell.delete_prefix('v:')) }
|
|
323
|
+
end
|
|
324
|
+
end
|
|
325
|
+
|
|
326
|
+
# @param branches [String] {RANKED_MATCH} or {MATCH}
|
|
327
|
+
# @param query [Query]
|
|
328
|
+
# @return [String] +branches+ once per group, joined by +UNION+
|
|
329
|
+
def self.union(branches, query)
|
|
330
|
+
Array.new(query.groups.size) { |i| format(branches, group: "group#{i}", index: i) }.join(' UNION ')
|
|
331
|
+
end
|
|
332
|
+
private_class_method :union
|
|
333
|
+
|
|
334
|
+
# @param query [Query]
|
|
335
|
+
# @return [Hash{Symbol => String}] +group0:+, +group1:+, … bound to each
|
|
336
|
+
# group's words, which +fts:match+ ANDs
|
|
337
|
+
def self.group_args(query)
|
|
338
|
+
query.groups.each_with_index.to_h { |words, i| [:"group#{i}", words.join(' ')] }
|
|
339
|
+
end
|
|
340
|
+
private_class_method :group_args
|
|
341
|
+
|
|
342
|
+
# @param count [Integer] how many folders the filter names
|
|
343
|
+
# @return [String] a SPARQL condition true for a URL under any of
|
|
344
|
+
# +~folder0+ … +~folder<count-1>+
|
|
345
|
+
def self.under(count)
|
|
346
|
+
Array.new(count) { |i| "STRSTARTS(?url, ~folder#{i})" }.join(' || ')
|
|
347
|
+
end
|
|
348
|
+
private_class_method :under
|
|
349
|
+
|
|
350
|
+
# @param folders [Array<String>]
|
|
351
|
+
# @return [Hash{Symbol => String}] +folder0:+, +folder1:+, … bound to
|
|
352
|
+
# each folder's URL prefix
|
|
353
|
+
def self.folder_args(folders)
|
|
354
|
+
folders.each_with_index.to_h { |folder, i| [:"folder#{i}", prefix(folder)] }
|
|
355
|
+
end
|
|
356
|
+
private_class_method :folder_args
|
|
357
|
+
|
|
358
|
+
# @param folder [String] absolute path
|
|
359
|
+
# @return [String] the URL prefix of every file below +folder+; the
|
|
360
|
+
# trailing slash keeps +/a/b+ from matching +/a/bc+
|
|
361
|
+
def self.prefix(folder)
|
|
362
|
+
"#{file_uri(folder).chomp('/')}/"
|
|
363
|
+
end
|
|
364
|
+
private_class_method :prefix
|
|
365
|
+
|
|
366
|
+
# @param query [String] SPARQL text
|
|
367
|
+
# @param args [Hash{Symbol => String, Integer}] the query's +~name+
|
|
368
|
+
# parameters
|
|
369
|
+
# @return [String] stdout
|
|
370
|
+
# @raise [CommandError] on a non-zero exit
|
|
371
|
+
# @raise [Errno::ENOENT] if tinysparql is missing
|
|
372
|
+
def self.run(query, **args)
|
|
373
|
+
argv = [BINARY, 'query', '-b', DBUS_SERVICE]
|
|
374
|
+
args.each do |name, value|
|
|
375
|
+
type = value.is_a?(Integer) ? 'i' : 's'
|
|
376
|
+
argv.push('--arg', "#{name}:#{type}:#{value}")
|
|
377
|
+
end
|
|
378
|
+
argv.push('-q', query)
|
|
379
|
+
result = Pikuri::Subprocess.spawn(*argv, chdir: '/').wait
|
|
380
|
+
return result.output if result.status.success?
|
|
381
|
+
|
|
382
|
+
message = result.output.strip
|
|
383
|
+
raise CommandError, message.empty? ? "exited #{result.status.exitstatus}" : message
|
|
384
|
+
end
|
|
385
|
+
private_class_method :run
|
|
386
|
+
|
|
387
|
+
# @param url [String] a +file://+ URI from the index
|
|
388
|
+
# @return [String] the absolute path it names
|
|
389
|
+
def self.path_of(url)
|
|
390
|
+
percent_decode(url.delete_prefix('file://'))
|
|
391
|
+
end
|
|
392
|
+
private_class_method :path_of
|
|
393
|
+
|
|
394
|
+
# @param encoded [String]
|
|
395
|
+
# @return [String] +%XX+ decoded per byte, tagged UTF-8
|
|
396
|
+
def self.percent_decode(encoded)
|
|
397
|
+
encoded.b.gsub(/%([0-9A-Fa-f]{2})/n) { ::Regexp.last_match(1).hex.chr }.force_encoding('UTF-8')
|
|
398
|
+
end
|
|
399
|
+
private_class_method :percent_decode
|
|
400
|
+
end
|
|
401
|
+
end
|
|
402
|
+
end
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pikuri
|
|
4
|
+
module Fileindex
|
|
5
|
+
# What the model typed into +fileindex_search+, as groups of words. A
|
|
6
|
+
# file matches when it holds every word of at least one group. A bare
|
|
7
|
+
# word is a group of its own; double quotes make one group of their
|
|
8
|
+
# words, in any order:
|
|
9
|
+
#
|
|
10
|
+
# q = Query.parse('"notice period" tenant lessee')
|
|
11
|
+
# q.groups # => [["notice", "period"], ["tenant"], ["lessee"]]
|
|
12
|
+
# q.all_of.groups # => [["notice", "period", "tenant", "lessee"]]
|
|
13
|
+
#
|
|
14
|
+
# Any group matching is the point: a model lists alternatives (synonyms,
|
|
15
|
+
# a second language) in one query, and an index that wants every word
|
|
16
|
+
# found nothing for half its queries (measured 2026-11). Quotes are a
|
|
17
|
+
# group, not a phrase: looser, so a model expecting a phrase still gets
|
|
18
|
+
# every file a phrase would find.
|
|
19
|
+
#
|
|
20
|
+
# Dropped while parsing: a bare +OR+ or +AND+ (written out of web-search
|
|
21
|
+
# habit, they would match nearly every file as words), a token with no
|
|
22
|
+
# letter or digit, and parentheses, since only quotes group. A quote left
|
|
23
|
+
# open runs to the end.
|
|
24
|
+
#
|
|
25
|
+
# Immutable.
|
|
26
|
+
Query = Data.define(:groups) do
|
|
27
|
+
# @param text [String] the model's query
|
|
28
|
+
# @return [Query] with no groups when +text+ holds no word
|
|
29
|
+
def self.parse(text)
|
|
30
|
+
groups = text.to_s.split('"', -1).each_with_index.flat_map do |part, i|
|
|
31
|
+
words = part.delete('()').split.reject { |word| word.match?(/\A(?:or|and)\z/i) || !word.match?(/[[:alnum:]]/) }
|
|
32
|
+
i.odd? ? [words] : words.map { |word| [word] }
|
|
33
|
+
end
|
|
34
|
+
new(groups: groups.reject(&:empty?).uniq)
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
# @param groups [Array<Array<String>>] each group's words, none empty
|
|
38
|
+
def initialize(groups:)
|
|
39
|
+
super(groups: groups.map { |words| words.dup.freeze }.freeze)
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
# @return [Boolean] whether there is no word to search for
|
|
43
|
+
def empty? = groups.empty?
|
|
44
|
+
|
|
45
|
+
# @return [Query] one group holding every word: the files that match
|
|
46
|
+
# the whole query, not just some of it
|
|
47
|
+
def all_of = Query.new(groups: [groups.flatten.uniq])
|
|
48
|
+
end
|
|
49
|
+
end
|
|
50
|
+
end
|