pikuri-fileindex 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 4d05f84868f8fc4907f3a16f3ec7e5c8a4bd6b57f476b49ad4abe0bb63d0def4
4
+ data.tar.gz: af99d134e391f4ba46a76ec48c6650b1afb6d26e00a3b93157b404262ef0b832
5
+ SHA512:
6
+ metadata.gz: 7d56e04cd9c67708fd2c366fa833c64424eea69dacfa84909e3b1f4e610de9cfa4e2faef55ab8ae97ee6aa059324f14695c902f6961636caf41f001b1a455553
7
+ data.tar.gz: 1931986bd6fe2ebcb2cd552994f720236d1b582b45022a3a35dea2d2d9e935444a15349d396a1c13e3f0b38d2e742a258d193eb4898217cb25a182a6c1e20726
data/README.md ADDED
@@ -0,0 +1,114 @@
1
+ # pikuri-fileindex
2
+
3
+ Search your files through the index your desktop already keeps, for the
4
+ [pikuri](https://github.com/mvysny/pikuri) AI-assistant toolkit. Ask
5
+ "find the Kettle & Co. invoice" and the agent gets back the PDF's path
6
+ and a 40-word snippet with the amount in it, without pikuri ever opening
7
+ the PDF itself.
8
+
9
+ Provides:
10
+ - `fileindex_search` (`Pikuri::Fileindex::Search`): keyword search, one
11
+ snippet per file (with localsearch, a file of up to 600 characters comes
12
+ whole), optionally limited to one folder. A file matches when
13
+ it holds any of the words (`rent lease vuokra`), or all the words in
14
+ double quotes, in any order; files with every word come first. It says
15
+ how many files match in all ("Showing 10 of 37"), so "all shown" means
16
+ there is no other.
17
+ - `fileindex_read` (`Pikuri::Fileindex::Read`): a file's full text as the
18
+ index extracted it, paged like `read`, with an optional `pattern` that
19
+ returns only the matching lines.
20
+ - `Pikuri::Fileindex::Roots`: the folders you let the agent reach. The
21
+ agent can't search or read anything outside them, or any hidden file
22
+ inside them.
23
+ - Two backends: GNOME's **localsearch** (queried through `tinysparql`)
24
+ and **recoll** (through `recollq`). `Pikuri::Fileindex.pick` takes the
25
+ first one that is ready.
26
+
27
+ ## Why borrow the desktop's index
28
+
29
+ - **No GPU bill, no daemon of ours.** GNOME already indexes your files
30
+ and keeps the index fresh as you save. Building our own embedding
31
+ index of `~/Documents` kept a laptop GPU at 100% for hours.
32
+ - **The desktop parses the hostile files, not pikuri.** localsearch
33
+ extracts the text of a `.docx` or a PDF in its own sandboxed process.
34
+ A malicious file in Downloads is never parsed by pikuri.
35
+ - **Office documents come free**: the index already holds the text of
36
+ `.docx`, `.odt` and spreadsheets (on Ubuntu, once
37
+ `localsearch-extractor-office` is installed).
38
+ - **Keyword search can say "that's every match"**, which a vector search
39
+ can't.
40
+
41
+ It never builds, configures or refreshes an index; it only reads one.
42
+
43
+ ## Requirements
44
+
45
+ One of:
46
+ - **localsearch 3.8 or newer** with the `tinysparql` CLI (Ubuntu 26.04
47
+ LTS and newer; Debian/Ubuntu's `localsearch` package pulls in
48
+ `tinysparql`). Older Tracker 3 (`tracker3`) is not supported. On Ubuntu,
49
+ also install `localsearch-extractor-office`: without it, `.docx`, `.odt`,
50
+ `.xlsx` and epub files are found by name only, never by their text. Files
51
+ indexed before you install it stay that way, even after a restart
52
+ ([a localsearch bug](https://gitlab.gnome.org/GNOME/localsearch/-/work_items/476)),
53
+ so run `localsearch reset` once afterwards: it erases the index and
54
+ localsearch builds it again, office text included. That takes a while;
55
+ `localsearch status` says when it's done.
56
+ - **recoll** with an index you built (`recollindex`). Run
57
+ `recollindex -m` to keep it fresh as files change.
58
+
59
+ What the index doesn't cover, the agent can't find: localsearch skips
60
+ hidden folders, anything inside a git checkout, and the text of source
61
+ code and `.csv` files; recoll covers only the folders in its `topdirs`.
62
+
63
+ ## Install
64
+
65
+ ```ruby
66
+ # Gemfile
67
+ gem 'pikuri-fileindex'
68
+ ```
69
+
70
+ ## Usage
71
+
72
+ ```ruby
73
+ require 'pikuri-core'
74
+ require 'pikuri-fileindex'
75
+
76
+ roots = Pikuri::Fileindex::Roots.new(dirs: %w[~/Desktop ~/Documents], private: true)
77
+ backend = Pikuri::Fileindex.pick(roots) # raises Pikuri::Fileindex::Unavailable, saying why
78
+ agent = Pikuri::Agent.new(transport:, system_prompt:) do |c|
79
+ c.add_extension Pikuri::Fileindex::Extension.new(roots:, backend:)
80
+ end
81
+ ```
82
+
83
+ **Name your folders.** A stock GNOME install indexes your whole home
84
+ folder: `~/work`, `~/snap`, your backups. Only the roots keep an agent
85
+ you gave `~/Documents` out of the rest.
86
+
87
+ `private:` says the folders hold data worth stealing; `trusted:` says
88
+ every file in them is your own, so nothing a stranger wrote (not
89
+ Downloads, not a folder an agent that reads your mail writes into).
90
+ Both default to `false`, and they set the tools' trifecta legs.
91
+
92
+ Pick once per process and hand the backend to every agent: checking
93
+ readiness runs one query per root. When neither index holds a file
94
+ under the roots, the `Unavailable` message names what is missing for
95
+ each index and how to fix it; show it to the user. An empty Desktop
96
+ among other roots is fine, and only logs a warning.
97
+
98
+ ## Try it
99
+
100
+ `bin/pikuri-briefing` prepares you for a customer meeting from your own
101
+ hand-written notes: the timeline, what was discussed last, what was
102
+ promised. The model runs locally and the agent has no internet access.
103
+
104
+ ```
105
+ ./pikuri-fileindex/bin/pikuri-briefing --source ~/notes "brief me on Acme"
106
+ ```
107
+
108
+ `bin/pikuri-secretary`, at the repository root, searches your Desktop
109
+ and Documents the same way.
110
+
111
+ ## Read more
112
+
113
+ Design notes and the backend survey:
114
+ [`design/architecture.md`](design/architecture.md).
@@ -0,0 +1,36 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Pikuri
4
+ module Fileindex
5
+ # Wires +fileindex_search+ and +fileindex_read+ over one backend and one
6
+ # set of roots, so the two tools can never disagree about which folders
7
+ # the agent reaches:
8
+ #
9
+ # roots = Pikuri::Fileindex::Roots.new(dirs: %w[~/Desktop ~/Documents], private: true)
10
+ # backend = Pikuri::Fileindex.pick(roots) # once per process; raises Unavailable
11
+ # Pikuri::Agent.new(transport:, system_prompt:) do |c|
12
+ # c.add_extension Pikuri::Fileindex::Extension.new(roots:, backend:)
13
+ # end
14
+ #
15
+ # The backend comes in rather than being picked here: picking probes the
16
+ # index once per root, and an extension is built anew for every agent.
17
+ class Extension
18
+ include Pikuri::Agent::Extension
19
+
20
+ # @param roots [Roots] the folders both tools reach
21
+ # @param backend [Module] a ready backend, from {Fileindex.pick}
22
+ def initialize(roots:, backend:)
23
+ @roots = roots
24
+ @backend = backend
25
+ end
26
+
27
+ # @param c [Pikuri::Agent::Configurator]
28
+ # @return [void]
29
+ def configure(c)
30
+ c.add_tool Search.new(backend: @backend, roots: @roots)
31
+ c.add_tool Read.new(backend: @backend, roots: @roots)
32
+ nil
33
+ end
34
+ end
35
+ end
36
+ end
@@ -0,0 +1,402 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Pikuri
4
+ module Fileindex
5
+ # The GNOME desktop file index (+localsearch+, formerly tracker-miners),
6
+ # queried in SPARQL through the +tinysparql+ CLI. The stateless backend
7
+ # behind +fileindex_search+ / +fileindex_read+:
8
+ #
9
+ # LocalSearch.search(query: Query.parse('lease rent'), limit: 10, folders: ['/home/u/Documents'])
10
+ # # => [{ path: '/home/u/Documents/Housing/lease-agreement.pdf',
11
+ # # snippet: 'Residential >>Lease<< Agreement Landlord: …',
12
+ # # text_size: 4211, text: nil },
13
+ # # { path: '/home/u/Documents/Notes/flat.md',
14
+ # # snippet: '# Flat The >>rent<< is EUR 1,150 …',
15
+ # # text_size: 212, text: "# Flat\nThe rent is EUR 1,150, due on the 5th.\n…" }, …]
16
+ # LocalSearch.read_text('/home/u/Documents/Housing/lease-agreement.pdf')
17
+ # # => "Residential Lease Agreement\nLandlord: …" (nil when not indexed)
18
+ #
19
+ # A file matches when it holds every word of any one of the query's
20
+ # groups, by its name or its extracted text. Hits come one per file,
21
+ # those matching more groups first, then by relevance, each with one
22
+ # snippet of about {SNIPPET_WORDS} words around a match, matched words
23
+ # marked +>>like this<<+, and a short file's whole text too (up to
24
+ # {WHOLE_TEXT_CHARS}). PDFs and office docs come for free: the index
25
+ # extracted their text, so {.read_text} needs no extractor here.
26
+ #
27
+ # Needs +localsearch+ (the indexing daemon, a GNOME default) and the
28
+ # +tinysparql+ CLI, which Debian/Ubuntu's +localsearch+ package depends on.
29
+ #
30
+ # == Implementation details
31
+ #
32
+ # Why SPARQL and not +localsearch search+: that CLI sorts its hits by
33
+ # path, not relevance, before applying the limit, and cannot filter by
34
+ # folder — so a common word returned the first ten paths A–Z under +$HOME+.
35
+ #
36
+ # +tinysparql query+ prints a row as its cells joined by +", "+, which
37
+ # snippets and paths also contain. So every cell leaves SPARQL as
38
+ # +"v:" + ENCODE_FOR_URI(value)+: no comma, space or newline survives
39
+ # encoding, and the +v:+ marker sets data rows apart from the CLI's
40
+ # +Results:+ / +None+ lines, which are translated by locale.
41
+ #
42
+ # Index URLs use GLib's escaping ({.file_uri}); a folder filter or a read
43
+ # built with any other escaping silently matches nothing.
44
+ #
45
+ # Each group and each folder is its own bound parameter (+~group0+,
46
+ # +~folder0+, …); only those generated names are spliced into the query
47
+ # text, never a word or a path. A parameter's type is tinysparql's
48
+ # one-letter code (+s+, +i+): +string+ and +xsd:string+ are refused.
49
+ #
50
+ # Any-group matching is a +UNION+ of one +fts:match+ per group, because
51
+ # +fts:match+ has no OR: it reads +OR+, +NOT+ and parentheses as words.
52
+ module LocalSearch
53
+ # @return [String] the SPARQL query CLI.
54
+ BINARY = 'tinysparql'
55
+
56
+ # @return [String] the D-Bus name the localsearch daemon serves its
57
+ # index under; querying it starts the daemon through D-Bus activation.
58
+ DBUS_SERVICE = 'org.freedesktop.LocalSearch3'
59
+
60
+ # Not raised to show more: once a file's whole text fits the window,
61
+ # +fts:snippet+ may answer its title alone (measured: a 46-word
62
+ # invoice matching "invoice" gave +Kettle & Co. — >>Invoice<< #4471+
63
+ # at 55 words). {WHOLE_TEXT_CHARS} shows short files instead.
64
+ #
65
+ # @return [Integer] words of context in a snippet: enough to tell an
66
+ # invoice from a book that mentions one without reading either.
67
+ SNIPPET_WORDS = 40
68
+
69
+ # @return [Integer] longest stored text, in characters, that a hit
70
+ # carries whole: a few sentences, about the 500 characters a vector
71
+ # search's chunk shows. A snippet of a file this short leaves the
72
+ # rest, often the answer (an invoice's total), to a read call: on a
73
+ # 212-file test home (2026-11, 90 lookups, every answer in a short
74
+ # file), showing it whole cut the reads after a search from 73 to 10,
75
+ # for 13% more text a search.
76
+ WHOLE_TEXT_CHARS = 600
77
+
78
+ # @return [Array(String, String)] what wraps each matched word in a
79
+ # snippet, so the model sees which words hit.
80
+ MATCH_MARKERS = ['>>', '<<'].freeze
81
+
82
+ # @return [Logger]
83
+ LOGGER = Pikuri.logger_for('Fileindex::LocalSearch')
84
+
85
+ # The ranked, folder-scoped search: +%<match>s+ is {RANKED_MATCH} once
86
+ # per group, joined by +UNION+, and +%<under>s+ the folder filter,
87
+ # {.under}'s. +MAX+ over a file's rows keeps the better rank and the
88
+ # non-empty snippet. The +OPTIONAL+ measures the file's stored text
89
+ # and keeps it if short, whichever branch matched; a file indexed by
90
+ # name only leaves both unbound.
91
+ #
92
+ # @return [String]
93
+ SEARCH_QUERY = <<~SPARQL
94
+ SELECT (CONCAT("v:", ENCODE_FOR_URI(?url)) AS ?u)
95
+ (CONCAT("v:", ENCODE_FOR_URI(MAX(?snippet))) AS ?s)
96
+ (CONCAT("v:", STR(MAX(?length))) AS ?n)
97
+ (CONCAT("v:", ENCODE_FOR_URI(MAX(?short))) AS ?t)
98
+ WHERE {
99
+ %<match>s
100
+ ?f nie:url ?url .
101
+ OPTIONAL {
102
+ ?stored nie:isStoredAs ?f ;
103
+ nie:plainTextContent ?text .
104
+ BIND (STRLEN(?text) AS ?length)
105
+ BIND (IF(?length <= #{WHOLE_TEXT_CHARS}, ?text, "") AS ?short)
106
+ }
107
+ FILTER (tracker:available(nie:dataSource(?f)))
108
+ FILTER (%<under>s)
109
+ }
110
+ GROUP BY ?url
111
+ ORDER BY DESC(COUNT(DISTINCT ?group)) DESC(MAX(?rank)) ?url
112
+ LIMIT ~limit
113
+ SPARQL
114
+
115
+ # One group's branches in {SEARCH_QUERY}: the file matches by name
116
+ # (first branch, no snippet) or by extracted text (second). +%<group>s+
117
+ # is the group's parameter name, +%<index>d+ its number, which ranks a
118
+ # file by how many groups it matched.
119
+ #
120
+ # @return [String]
121
+ RANKED_MATCH = <<~SPARQL.chomp
122
+ {
123
+ ?f fts:match ~%<group>s .
124
+ BIND (fts:rank(?f) AS ?rank)
125
+ BIND ("" AS ?snippet)
126
+ BIND (%<index>d AS ?group)
127
+ } UNION {
128
+ ?elem fts:match ~%<group>s ;
129
+ nie:isStoredAs ?f .
130
+ BIND (fts:rank(?elem) AS ?rank)
131
+ BIND (fts:snippet(?elem, "#{MATCH_MARKERS[0]}", "#{MATCH_MARKERS[1]}", "…", #{SNIPPET_WORDS}) AS ?snippet)
132
+ BIND (%<index>d AS ?group)
133
+ }
134
+ SPARQL
135
+
136
+ # Every file {SEARCH_QUERY} matches, unranked and uncapped: the same
137
+ # branches ({MATCH}) and filters, no rank or snippet to compute.
138
+ #
139
+ # @return [String]
140
+ PATHS_QUERY = <<~SPARQL
141
+ SELECT DISTINCT (CONCAT("v:", ENCODE_FOR_URI(?url)) AS ?u)
142
+ WHERE {
143
+ %<match>s
144
+ ?f nie:url ?url .
145
+ FILTER (tracker:available(nie:dataSource(?f)))
146
+ FILTER (%<under>s)
147
+ }
148
+ SPARQL
149
+
150
+ # One group's branches in {PATHS_QUERY}.
151
+ #
152
+ # @return [String]
153
+ MATCH = '{ ?f fts:match ~%<group>s . } UNION { ?elem fts:match ~%<group>s ; nie:isStoredAs ?f . }'
154
+
155
+ # @return [String] the stored text of the file at +~url+.
156
+ READ_QUERY = <<~SPARQL
157
+ SELECT (CONCAT("v:", ENCODE_FOR_URI(?text)) AS ?t)
158
+ WHERE {
159
+ ?f nie:url ~url .
160
+ ?elem nie:isStoredAs ?f ;
161
+ nie:plainTextContent ?text .
162
+ }
163
+ LIMIT 1
164
+ SPARQL
165
+
166
+ # Any one searchable file under +~folder0+: a folder alone gives a
167
+ # search nothing to find, and a file on an unmounted drive is filtered
168
+ # as {SEARCH_QUERY} filters it.
169
+ #
170
+ # @return [String]
171
+ PROBE_QUERY = <<~SPARQL
172
+ SELECT (CONCAT("v:", ENCODE_FOR_URI(?url)) AS ?u)
173
+ WHERE {
174
+ ?f nie:url ?url .
175
+ ?elem nie:isStoredAs ?f .
176
+ FILTER NOT EXISTS { ?elem a nfo:Folder }
177
+ FILTER (tracker:available(nie:dataSource(?f)))
178
+ FILTER (STRSTARTS(?url, ~folder0))
179
+ }
180
+ LIMIT 1
181
+ SPARQL
182
+
183
+ # @return [Regexp] a byte GLib leaves unescaped in a +file://+ URI
184
+ # (+g_filename_to_uri+'s set, measured against the index); anything
185
+ # else, non-ASCII included, is +%XX+ per byte.
186
+ URI_SAFE = %r{[A-Za-z0-9\-._~!$&'()*+,=:@/]}
187
+
188
+ # Raised when a tinysparql invocation exits non-zero — a recoverable
189
+ # failure the calling tool renders as +"Error: ..."+.
190
+ class CommandError < StandardError; end
191
+
192
+ # @return [String] short human name for this backend, used by the
193
+ # fileindex tools when prefixing an +"Error: ..."+ observation.
194
+ def self.label = 'localsearch'
195
+
196
+ # Uniform accessor mirroring {Recoll.limitations}; a constant here
197
+ # because this index's blind spots are the same on every host.
198
+ #
199
+ # @return [String]
200
+ def self.limitations = LIMITATIONS
201
+
202
+ # What this index will not surface, appended to the fileindex tools'
203
+ # LLM description so the model knows when to search the files directly
204
+ # instead. Measured; see +pikuri-fileindex/design/architecture.md+.
205
+ #
206
+ # @return [String]
207
+ LIMITATIONS = <<~LIMITS.chomp
208
+ Not covered by this index (fall back to a text/regex search over the files for these):
209
+ - Source code and other developer files are not content-indexed — only their names are. Search their text directly.
210
+ - Nothing inside a git checkout is indexed (any folder containing a .git). Search those files directly.
211
+ LIMITS
212
+
213
+ # Ranked full-text search under any of +folders+.
214
+ #
215
+ # @param query [Query] a file matches every word, stemmed, of any group
216
+ # @param limit [Integer] max files returned
217
+ # @param folders [Array<String>] absolute real paths of the folders to
218
+ # search under, subfolders included; at least one
219
+ # @return [Array<Hash>] +[{ path: String, snippet: String|nil,
220
+ # text_size: Integer|nil, text: String|nil }, …]+, best first, one
221
+ # per file; +text_size+ is the stored text's length in characters
222
+ # (SPARQL has no byte length; the two differ only by the non-ASCII
223
+ # share), +nil+ for a file indexed by name only; +text+ is the
224
+ # whole stored text when +text_size+ is at most {WHOLE_TEXT_CHARS}
225
+ # @raise [CommandError] if tinysparql exits non-zero
226
+ def self.search(query:, limit:, folders:)
227
+ raw = run(format(SEARCH_QUERY, match: union(RANKED_MATCH, query), under: under(folders.size)),
228
+ limit: limit, **group_args(query), **folder_args(folders))
229
+ parse_rows(raw).map do |url, snippet, length, text|
230
+ snippet = snippet&.gsub(/\s+/, ' ')&.strip
231
+ { path: path_of(url), snippet: (snippet unless snippet.to_s.empty?), text_size: length&.to_i,
232
+ text: (text unless text.to_s.strip.empty?) }
233
+ end
234
+ end
235
+
236
+ # The path of every file {.search} would match, unranked and uncapped,
237
+ # so a caller can tell how many matched beyond its +limit+.
238
+ #
239
+ # @param query [Query]
240
+ # @param folders [Array<String>] as {.search}'s
241
+ # @return [Array<String>] absolute paths
242
+ # @raise [CommandError] if tinysparql exits non-zero
243
+ def self.paths(query:, folders:)
244
+ raw = run(format(PATHS_QUERY, match: union(MATCH, query), under: under(folders.size)),
245
+ **group_args(query), **folder_args(folders))
246
+ parse_rows(raw).map { |(url)| path_of(url) }
247
+ end
248
+
249
+ # Full stored plain text of an indexed file. Works for PDFs/office docs
250
+ # (extracted at index time).
251
+ #
252
+ # @param path [String] absolute path
253
+ # @return [String, nil] +nil+ when the path isn't indexed, has no stored
254
+ # text, or the query fails — the caller turns that into an LLM-facing
255
+ # message
256
+ def self.read_text(path)
257
+ row = parse_rows(run(READ_QUERY, url: file_uri(File.expand_path(path)))).first
258
+ text = row&.first
259
+ text && !text.strip.empty? ? text : nil
260
+ rescue CommandError
261
+ nil
262
+ end
263
+
264
+ # Why this index can't answer a search under +folders+ right now, for
265
+ # a human to act on; +nil+ when it can. An index that answers but holds
266
+ # no file under any of them is not ready either: every search would
267
+ # come back empty. Some empty folders are normal (an empty Desktop), so
268
+ # they only log a warning naming each.
269
+ #
270
+ # LocalSearch.unavailable_reason(folders: ['/home/u/notes'])
271
+ # # => "nothing under /home/u/notes is indexed. …" (nil when ready)
272
+ #
273
+ # Starts the localsearch daemon if it is installed but not running, as
274
+ # any query does (D-Bus activation).
275
+ #
276
+ # @param folders [Array<String>] absolute real paths
277
+ # @return [String, nil]
278
+ def self.unavailable_reason(folders:)
279
+ empty = folders.reject { |folder| parse_rows(run(PROBE_QUERY, folder0: prefix(folder))).any? }
280
+ if empty.size < folders.size
281
+ empty.each { |folder| LOGGER.warn("nothing under #{folder} is indexed; searches there find nothing") }
282
+ return nil
283
+ end
284
+
285
+ "nothing under #{folders.join(', ')} is indexed. Right after a first login GNOME may still be " \
286
+ 'indexing: `localsearch status` shows how far it got. Otherwise add the folders under ' \
287
+ 'GNOME Settings → Search. A folder inside a git checkout is never indexed.'
288
+ rescue Errno::ENOENT
289
+ "the '#{BINARY}' command is missing. Install the 'localsearch' and '#{BINARY}' " \
290
+ "packages: GNOME's desktop file index and its query tool."
291
+ rescue CommandError => e
292
+ "the index doesn't answer (#{e.message}). Is the 'localsearch' package " \
293
+ "installed? It is GNOME's desktop file indexer."
294
+ end
295
+
296
+ # The +file://+ URI the index stores for +path+, escaped the way GLib
297
+ # escapes it.
298
+ #
299
+ # LocalSearch.file_uri('/home/u/My Notes/ü#1.txt')
300
+ # # => "file:///home/u/My%20Notes/%C3%BC%231.txt"
301
+ #
302
+ # @param path [String] absolute path
303
+ # @return [String]
304
+ def self.file_uri(path)
305
+ escaped = path.b.gsub(/./mn) { |c| c.match?(URI_SAFE) ? c : format('%%%02X', c.ord) }
306
+ "file://#{escaped}"
307
+ end
308
+
309
+ # Parse +tinysparql query+ output into rows of decoded cells. Only rows
310
+ # whose every cell carries the +v:+ marker count (see the class doc).
311
+ #
312
+ # @param raw [String] the CLI's stdout
313
+ # @return [Array<Array<String, nil>>] one array of cells per row; an
314
+ # unbound cell (+(null)+) is +nil+
315
+ def self.parse_rows(raw)
316
+ raw.each_line(chomp: true).filter_map do |line|
317
+ next unless line.start_with?(' ')
318
+
319
+ cells = line.strip.split(', ')
320
+ next unless cells.all? { |cell| cell == '(null)' || cell.start_with?('v:') }
321
+
322
+ cells.map { |cell| cell == '(null)' ? nil : percent_decode(cell.delete_prefix('v:')) }
323
+ end
324
+ end
325
+
326
+ # @param branches [String] {RANKED_MATCH} or {MATCH}
327
+ # @param query [Query]
328
+ # @return [String] +branches+ once per group, joined by +UNION+
329
+ def self.union(branches, query)
330
+ Array.new(query.groups.size) { |i| format(branches, group: "group#{i}", index: i) }.join(' UNION ')
331
+ end
332
+ private_class_method :union
333
+
334
+ # @param query [Query]
335
+ # @return [Hash{Symbol => String}] +group0:+, +group1:+, … bound to each
336
+ # group's words, which +fts:match+ ANDs
337
+ def self.group_args(query)
338
+ query.groups.each_with_index.to_h { |words, i| [:"group#{i}", words.join(' ')] }
339
+ end
340
+ private_class_method :group_args
341
+
342
+ # @param count [Integer] how many folders the filter names
343
+ # @return [String] a SPARQL condition true for a URL under any of
344
+ # +~folder0+ … +~folder<count-1>+
345
+ def self.under(count)
346
+ Array.new(count) { |i| "STRSTARTS(?url, ~folder#{i})" }.join(' || ')
347
+ end
348
+ private_class_method :under
349
+
350
+ # @param folders [Array<String>]
351
+ # @return [Hash{Symbol => String}] +folder0:+, +folder1:+, … bound to
352
+ # each folder's URL prefix
353
+ def self.folder_args(folders)
354
+ folders.each_with_index.to_h { |folder, i| [:"folder#{i}", prefix(folder)] }
355
+ end
356
+ private_class_method :folder_args
357
+
358
+ # @param folder [String] absolute path
359
+ # @return [String] the URL prefix of every file below +folder+; the
360
+ # trailing slash keeps +/a/b+ from matching +/a/bc+
361
+ def self.prefix(folder)
362
+ "#{file_uri(folder).chomp('/')}/"
363
+ end
364
+ private_class_method :prefix
365
+
366
+ # @param query [String] SPARQL text
367
+ # @param args [Hash{Symbol => String, Integer}] the query's +~name+
368
+ # parameters
369
+ # @return [String] stdout
370
+ # @raise [CommandError] on a non-zero exit
371
+ # @raise [Errno::ENOENT] if tinysparql is missing
372
+ def self.run(query, **args)
373
+ argv = [BINARY, 'query', '-b', DBUS_SERVICE]
374
+ args.each do |name, value|
375
+ type = value.is_a?(Integer) ? 'i' : 's'
376
+ argv.push('--arg', "#{name}:#{type}:#{value}")
377
+ end
378
+ argv.push('-q', query)
379
+ result = Pikuri::Subprocess.spawn(*argv, chdir: '/').wait
380
+ return result.output if result.status.success?
381
+
382
+ message = result.output.strip
383
+ raise CommandError, message.empty? ? "exited #{result.status.exitstatus}" : message
384
+ end
385
+ private_class_method :run
386
+
387
+ # @param url [String] a +file://+ URI from the index
388
+ # @return [String] the absolute path it names
389
+ def self.path_of(url)
390
+ percent_decode(url.delete_prefix('file://'))
391
+ end
392
+ private_class_method :path_of
393
+
394
+ # @param encoded [String]
395
+ # @return [String] +%XX+ decoded per byte, tagged UTF-8
396
+ def self.percent_decode(encoded)
397
+ encoded.b.gsub(/%([0-9A-Fa-f]{2})/n) { ::Regexp.last_match(1).hex.chr }.force_encoding('UTF-8')
398
+ end
399
+ private_class_method :percent_decode
400
+ end
401
+ end
402
+ end
@@ -0,0 +1,50 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Pikuri
4
+ module Fileindex
5
+ # What the model typed into +fileindex_search+, as groups of words. A
6
+ # file matches when it holds every word of at least one group. A bare
7
+ # word is a group of its own; double quotes make one group of their
8
+ # words, in any order:
9
+ #
10
+ # q = Query.parse('"notice period" tenant lessee')
11
+ # q.groups # => [["notice", "period"], ["tenant"], ["lessee"]]
12
+ # q.all_of.groups # => [["notice", "period", "tenant", "lessee"]]
13
+ #
14
+ # Any group matching is the point: a model lists alternatives (synonyms,
15
+ # a second language) in one query, and an index that wants every word
16
+ # found nothing for half its queries (measured 2026-11). Quotes are a
17
+ # group, not a phrase: looser, so a model expecting a phrase still gets
18
+ # every file a phrase would find.
19
+ #
20
+ # Dropped while parsing: a bare +OR+ or +AND+ (written out of web-search
21
+ # habit, they would match nearly every file as words), a token with no
22
+ # letter or digit, and parentheses, since only quotes group. A quote left
23
+ # open runs to the end.
24
+ #
25
+ # Immutable.
26
+ Query = Data.define(:groups) do
27
+ # @param text [String] the model's query
28
+ # @return [Query] with no groups when +text+ holds no word
29
+ def self.parse(text)
30
+ groups = text.to_s.split('"', -1).each_with_index.flat_map do |part, i|
31
+ words = part.delete('()').split.reject { |word| word.match?(/\A(?:or|and)\z/i) || !word.match?(/[[:alnum:]]/) }
32
+ i.odd? ? [words] : words.map { |word| [word] }
33
+ end
34
+ new(groups: groups.reject(&:empty?).uniq)
35
+ end
36
+
37
+ # @param groups [Array<Array<String>>] each group's words, none empty
38
+ def initialize(groups:)
39
+ super(groups: groups.map { |words| words.dup.freeze }.freeze)
40
+ end
41
+
42
+ # @return [Boolean] whether there is no word to search for
43
+ def empty? = groups.empty?
44
+
45
+ # @return [Query] one group holding every word: the files that match
46
+ # the whole query, not just some of it
47
+ def all_of = Query.new(groups: [groups.flatten.uniq])
48
+ end
49
+ end
50
+ end