pikuri-pdf 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +5 -7
- data/lib/pikuri/extractors/pdf.rb +4 -4
- metadata +8 -8
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 34536305f76e5f764bfb0dac7d8e72d07adbc797ca2356a6043feef42ec491d9
|
|
4
|
+
data.tar.gz: b4e13b3d7adac3be0fea7fceb37ce49335eece04b9b76781b453cc48dd70e754
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: d7fe20534136573f99c2684ebec40cfd323c24c2e0ee9ffaa0d2bd7ceb7174c0794b5531c76147ad2ad1593e81a45a4903e6d84df77367521c138a8c1408daaf
|
|
7
|
+
data.tar.gz: 83541604b7a55768ba61ab024226290330f762904854ce12a1b95258d9c0207ad98d2149cc0cb7ef5a0f6d7e5c38c288216f317e2db88adbe5ea352d2a556831
|
data/README.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# pikuri-pdf
|
|
2
2
|
|
|
3
3
|
PDF text extraction for the
|
|
4
|
-
[pikuri](https://
|
|
4
|
+
[pikuri](https://github.com/mvysny/pikuri) AI-assistant toolkit:
|
|
5
5
|
in-process, pure Ruby, and **lazy** — paged reads parse only the
|
|
6
6
|
pages the window needs, so showing the first page of a 500-page PDF
|
|
7
7
|
never pays for the other 499.
|
|
@@ -13,8 +13,7 @@ Provides:
|
|
|
13
13
|
registered, every pikuri surface that routes through the registry
|
|
14
14
|
picks PDFs up for free: the `read` tool pages through a local
|
|
15
15
|
`.pdf` with `--- Page N ---` markers, `web_scrape` extracts a
|
|
16
|
-
downloaded paper,
|
|
17
|
-
corpus.
|
|
16
|
+
downloaded paper, a PDF mail attachment reads as text.
|
|
18
17
|
|
|
19
18
|
## Why a separate gem
|
|
20
19
|
|
|
@@ -28,7 +27,7 @@ needs PDFs, skip it (and its whole subtree) when it doesn't.
|
|
|
28
27
|
Everything is pure Ruby, so the worst a malicious PDF can do to the
|
|
29
28
|
parser is burn CPU and memory — there's no native code to corrupt.
|
|
30
29
|
|
|
31
|
-
**This gem or [pikuri-extractors](
|
|
30
|
+
**This gem or [pikuri-extractors](https://github.com/mvysny/pikuri-workshop/blob/master/pikuri-extractors) — pick one
|
|
32
31
|
per wiring.** pikuri-extractors' converter container also has a PDF
|
|
33
32
|
arm (poppler's `pdftotext`, sandboxed, same `--- Page N ---`
|
|
34
33
|
markers): on an agent that fetches untrusted documents from the
|
|
@@ -37,9 +36,8 @@ posture. This gem is the no-infrastructure wiring — in-process
|
|
|
37
36
|
means no docker and no host CLIs, and it's what makes the lazy
|
|
38
37
|
page-windowed reads possible (a subprocess converter must convert
|
|
39
38
|
the whole document before emitting anything, and re-converts it on
|
|
40
|
-
every paged read). The guide wires this gem
|
|
41
|
-
|
|
42
|
-
universal-assistant chapter.
|
|
39
|
+
every paged read). The guide's vector-DB chapter wires this gem and
|
|
40
|
+
shows when to swap in pikuri-extractors.
|
|
43
41
|
|
|
44
42
|
## Install
|
|
45
43
|
|
|
@@ -7,7 +7,7 @@ module Pikuri
|
|
|
7
7
|
# PDF → text extractor over the +pdf-reader+ gem: each page emits a
|
|
8
8
|
# +"--- Page N ---"+ marker line then its text, blocks joined by
|
|
9
9
|
# single newlines. The markers carry page provenance downstream (a
|
|
10
|
-
# cited answer, a +
|
|
10
|
+
# cited answer, the page a +read+ window starts on). A page with no
|
|
11
11
|
# extractable text contributes nothing — no marker — so a fully
|
|
12
12
|
# scanned PDF extracts to the empty String, a deliberate silent
|
|
13
13
|
# skip callers detect by length. No OCR here: +pdf-reader+ gives
|
|
@@ -22,7 +22,7 @@ module Pikuri
|
|
|
22
22
|
# *lazily* ({.extract_lines} parses pages on demand) — a property a
|
|
23
23
|
# subprocess converter structurally cannot have. (The full
|
|
24
24
|
# this-gem-vs-pikuri-extractors PDF trade-off is in
|
|
25
|
-
# +pikuri-extractors/
|
|
25
|
+
# +pikuri-extractors/design/architecture.md+.)
|
|
26
26
|
#
|
|
27
27
|
# == Registration is explicit
|
|
28
28
|
#
|
|
@@ -30,8 +30,8 @@ module Pikuri
|
|
|
30
30
|
# +Pikuri::Extractors::PDF.register+, which *front*-inserts into the
|
|
31
31
|
# registry (unlike pikuri-extractors' before-terminal insert)
|
|
32
32
|
# because the +%PDF-+ magic sniff is the registry's strongest
|
|
33
|
-
# signal — it must beat +HTML+
|
|
34
|
-
# lying header still extracts, and it never misfires on text.
|
|
33
|
+
# signal — it must beat pikuri-web's +HTML+ content-type match so a
|
|
34
|
+
# PDF under a lying header still extracts, and it never misfires on text.
|
|
35
35
|
module PDF
|
|
36
36
|
# Insert this extractor at the front of
|
|
37
37
|
# +Pikuri::Extractor.registry+ (see "Registration is explicit"
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: pikuri-pdf
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Martin Vysny
|
|
@@ -15,14 +15,14 @@ dependencies:
|
|
|
15
15
|
requirements:
|
|
16
16
|
- - '='
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
|
-
version: 0.
|
|
18
|
+
version: 0.2.0
|
|
19
19
|
type: :runtime
|
|
20
20
|
prerelease: false
|
|
21
21
|
version_requirements: !ruby/object:Gem::Requirement
|
|
22
22
|
requirements:
|
|
23
23
|
- - '='
|
|
24
24
|
- !ruby/object:Gem::Version
|
|
25
|
-
version: 0.
|
|
25
|
+
version: 0.2.0
|
|
26
26
|
- !ruby/object:Gem::Dependency
|
|
27
27
|
name: pdf-reader
|
|
28
28
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -53,7 +53,7 @@ description: |
|
|
|
53
53
|
Registration is explicit — +Pikuri::Extractors::PDF.register+ — so
|
|
54
54
|
requiring the gem changes nothing by itself; the host script picks
|
|
55
55
|
which extractors it wires in. One registration extends the +read+
|
|
56
|
-
tool, +web_scrape
|
|
56
|
+
tool, +web_scrape+ and mail attachments at once.
|
|
57
57
|
email:
|
|
58
58
|
- martin@vysny.me
|
|
59
59
|
executables: []
|
|
@@ -63,13 +63,13 @@ files:
|
|
|
63
63
|
- README.md
|
|
64
64
|
- lib/pikuri-pdf.rb
|
|
65
65
|
- lib/pikuri/extractors/pdf.rb
|
|
66
|
-
homepage: https://
|
|
66
|
+
homepage: https://github.com/mvysny/pikuri
|
|
67
67
|
licenses:
|
|
68
68
|
- MIT
|
|
69
69
|
metadata:
|
|
70
|
-
source_code_uri: https://
|
|
71
|
-
changelog_uri: https://
|
|
72
|
-
bug_tracker_uri: https://
|
|
70
|
+
source_code_uri: https://github.com/mvysny/pikuri/tree/master
|
|
71
|
+
changelog_uri: https://github.com/mvysny/pikuri/blob/master/CHANGELOG.md
|
|
72
|
+
bug_tracker_uri: https://github.com/mvysny/pikuri/issues
|
|
73
73
|
rubygems_mfa_required: 'true'
|
|
74
74
|
rdoc_options: []
|
|
75
75
|
require_paths:
|