dbf 5.3.0 → 5.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +67 -2
- data/LICENSE +1 -1
- data/README.md +51 -39
- data/dbf.gemspec +5 -1
- data/lib/dbf/cli.rb +133 -14
- data/lib/dbf/column.rb +17 -4
- data/lib/dbf/column_builder.rb +15 -2
- data/lib/dbf/column_type.rb +10 -3
- data/lib/dbf/database/foxpro.rb +18 -6
- data/lib/dbf/errors.rb +14 -0
- data/lib/dbf/file_handler.rb +19 -8
- data/lib/dbf/header.rb +3 -0
- data/lib/dbf/memo/dbase3.rb +6 -1
- data/lib/dbf/memo/dbase4.rb +15 -1
- data/lib/dbf/memo/foxpro.rb +21 -16
- data/lib/dbf/record.rb +4 -4
- data/lib/dbf/record_context.rb +2 -1
- data/lib/dbf/record_iterator.rb +52 -11
- data/lib/dbf/schema.rb +7 -8
- data/lib/dbf/table.rb +68 -15
- data/lib/dbf/version.rb +1 -1
- data/lib/dbf/version_config.rb +14 -3
- data/lib/dbf.rb +1 -0
- metadata +11 -6
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ff36c9c3d6b7a558cd5bf0f703f64f0cb7b22cbf6b4c52b843c855328a712a0a
|
|
4
|
+
data.tar.gz: ac0f725043e7dc9102a32381f6041e54ebd852a67a264778ae58b2a59430b01e
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 269035188f9741d80575efca78ae0bda153c4f27c79dbe3f3e6221fc714be22eb2fb9ec39899dcfc9d489f76d60bfaf1f892d6eeaaeea80fa406f5f401e909d7
|
|
7
|
+
data.tar.gz: 6203ede51060bad9d193b15bb98d814aa870d99dd29d6eb8ddd4cb2b5e04f078f522fb7f3051befcbebe11be86a13307da38f9981bbcb704c5307ebdb16dd6a3
|
data/CHANGELOG.md
CHANGED
|
@@ -1,7 +1,72 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
##
|
|
4
|
-
|
|
3
|
+
## 5.5.0
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
- `DBF::Table.new` accepts any IO-like object responding to `#read` and `#seek` (`File`, `Tempfile`, …) in addition to a path or `StringIO`, matching what the README already documented; IO inputs are switched to binary mode, and memo auto-discovery works from the IO's path when it has one
|
|
8
|
+
- `DBF::Error` base class: all library errors (`FileNotFoundError`, `NoColumnsDefined`, `Column::LengthError`, `Column::InvalidNameError`) now inherit from it, so callers can `rescue DBF::Error` to catch anything the library raises
|
|
9
|
+
- `Table#filename` now works for any IO input with a path, not only `File`
|
|
10
|
+
- `DBF::Table.open`: same arguments as `.new`, but the block form yields the table, closes it when the block returns, and returns the block's value — the same contract as `File.open`
|
|
11
|
+
- CI: lint job running RuboCop and Reek on every push and pull request
|
|
12
|
+
- CI: test on Windows and macOS (Ruby 3.4) in addition to Linux, since much real-world DBF data originates on Windows/FoxPro systems
|
|
13
|
+
- CI: observe-only (non-blocking) test jobs for JRuby, TruffleRuby, and Ruby head
|
|
14
|
+
- CI: enforce a 99% minimum test coverage floor via SimpleCov
|
|
15
|
+
- Docs site (dbf.infused.org): YARD docs are now built and deployed by CI instead of being committed to the repository; the Pages workflow's deprecated v2/v3 actions (whose artifact backend was shut down in early 2025) are updated to current versions, repairing the broken deploy
|
|
16
|
+
- Dependabot updates for GitHub Actions and gems (weekly), a security policy (SECURITY.md) with private vulnerability reporting, and contributor documentation (CONTRIBUTING.md)
|
|
17
|
+
- Automated releases: pushing a `vX.Y.Z` tag now builds and publishes the gem to RubyGems.org from CI via Trusted Publishing (OIDC) — no long-lived API key. `bundler/gem_tasks` is loaded in the Rakefile, providing `rake build` / `rake release`. Git tags resume at v5.4.0; versions 4.0.0–5.3.0 were released without tags
|
|
18
|
+
|
|
19
|
+
- Ractor safety: all library constants are verified Ractor-shareable and reading/exporting tables inside non-main Ractors is covered by specs
|
|
20
|
+
- CLI: `-j` exports records as a JSON array and `-J` as JSON Lines (one record per line, streamed in constant memory); deleted records are skipped, control bytes are escaped by JSON itself, and binary cells are scrubbed instead of raising
|
|
21
|
+
|
|
22
|
+
### Fixed
|
|
23
|
+
|
|
24
|
+
- `Table#to_csv(path)` closes the file it opens, so the CSV is fully flushed to disk when the method returns (previously the data could stay buffered until garbage collection)
|
|
25
|
+
- Visual FoxPro: a `.dbc` container supplying fewer long names than the table has columns no longer crashes column building; missing long names fall back to the table's own column names. Column rebuilding is also no longer O(n²)
|
|
26
|
+
- `Table#schema` validates the format against the documented list up front; a `NoMethodError` raised inside a valid schema generator is no longer misreported as "not a valid schema"
|
|
27
|
+
- FoxPro memo files shorter than their 512-byte header no longer crash memo reads (they return nil like other truncations)
|
|
28
|
+
- RuboCop config: restore the default `vendor/**/*` exclusion (overriding `Exclude` replaces the defaults), fixing a CI lint crash where RuboCop descended into cached vendored gems and tried to load plugins from their configs
|
|
29
|
+
- CLI: close the table when finished, so the DBF file can be deleted or replaced immediately afterwards on Windows (an open handle blocks deletion there)
|
|
30
|
+
|
|
31
|
+
### Removed
|
|
32
|
+
|
|
33
|
+
- CodeClimate config and README badges: the service is sunset, and coverage is now enforced directly in CI
|
|
34
|
+
|
|
35
|
+
### Changed
|
|
36
|
+
|
|
37
|
+
- Streaming reads: `Table#each` now reads records in 4 MB chunks of whole records instead of loading the entire record section into memory, so enumerating multi-gigabyte files (for example shapefile `.dbf` sidecars) uses bounded memory. On a 190 MB file, peak process memory drops from ~222 MB to ~35 MB with identical results. No API change; all malformed-file bounds are preserved
|
|
38
|
+
- Internal: `DBF::RecordContext` is now an immutable `Data` class instead of a `Struct`
|
|
39
|
+
- Internal: remove a redundant `Memo::Foxpro#initialize` and the duplicate `Header::HEADER_SIZE` constant (now sourced from `VersionConfig`)
|
|
40
|
+
- README: replace the stack of version-support notes with a compatibility table, point API docs at dbf.infused.org, and refer to LICENSE instead of inlining the full MIT text; copyright years updated through 2026
|
|
41
|
+
- Gemspec: add `homepage_uri`, `bug_tracker_uri`, `documentation_uri`, and `funding_uri` metadata (shown on the rubygems.org sidebar), and constrain the `csv` runtime dependency to `~> 3.3`
|
|
42
|
+
- `DBF::Column::NameError` is renamed to `DBF::Column::InvalidNameError` (it shadowed Ruby's `::NameError` without being one); the old constant remains as a deprecated alias
|
|
43
|
+
- Rescues in date/datetime decoding and FoxPro memo reads are narrowed to the specific expected errors, so genuine programmer errors surface instead of silently returning nil
|
|
44
|
+
- Declare `reek` and `simplecov` as direct development dependencies instead of relying on rubycritic's transitive dependencies
|
|
45
|
+
- Restrict `debug` and `ruby-lsp` to MRI so `bundle install` succeeds on JRuby/TruffleRuby
|
|
46
|
+
- Test matrix no longer fails fast, and a Ruby head regression no longer fails the build
|
|
47
|
+
- Update pinned `ruby/setup-ruby` action to v1.321.0
|
|
48
|
+
|
|
49
|
+
## 5.4.0
|
|
50
|
+
|
|
51
|
+
- CLI: replace terminal control bytes in file-derived output so a crafted DBF cannot emit escape sequences to an interactive terminal; CSV and schema output are only filtered when writing to a terminal, so redirected exports are unchanged
|
|
52
|
+
- Security (CWE-248): write binary or invalidly encoded cells in `Table#to_csv` as representable text instead of raising an encoding error
|
|
53
|
+
- Security (CWE-248): replace unrepresentable column type bytes so a corrupt descriptor cannot raise when the schema is serialized to JSON
|
|
54
|
+
- Security (CWE-248): `Table#record` returns nil when the file ends after the delete flag instead of crashing on a nil record body
|
|
55
|
+
- Security (CWE-248): stop column parsing when the file ends mid-descriptor instead of raising `ArgumentError` from a short unpack
|
|
56
|
+
- Security (CWE-248): guard FoxPro memo-pointer decoding against a nil value from a truncated record instead of crashing on nil.unpack1
|
|
57
|
+
- Security (CWE-248): guard dBase IV memo reads against a nil/short block header instead of crashing on nil.unpack1
|
|
58
|
+
- Security (CWE-248): guard dBase III memo reads against a start block past EOF instead of crashing on a nil block
|
|
59
|
+
- Security (CWE-248): stop column parsing on a truncated descriptor instead of constructing an invalid column that crashes on a nil length
|
|
60
|
+
- Security (CWE-248): decode truncated numeric cells (Currency, AutoIncrement) to blank instead of raising an uncaught `nil` crash
|
|
61
|
+
- Security (CWE-248): handle truncated headers and missing column terminators gracefully instead of raising an uncaught `nil` crash during column parsing
|
|
62
|
+
- Security (CWE-400): bound FoxPro memo reads by the memo file size so a crafted memo size cannot force a ~4 GiB allocation
|
|
63
|
+
- Security (CWE-400): bound dBase IV memo reads by the memo file size so a crafted length field cannot force a ~4 GiB allocation
|
|
64
|
+
- Security (CWE-835): bound record iteration by the bytes actually read so a crafted `record_count` (or zero `record_length`) cannot cause an unbounded loop
|
|
65
|
+
- Security (CWE-789): bound the record read buffer by the file's actual size so a crafted header cannot force a multi-gigabyte allocation from a tiny file
|
|
66
|
+
- Security (CWE-400): resolve FoxPro `.dbc` tables by scanning the directory instead of `Dir.glob`, preventing glob brace-expansion CPU exhaustion from a crafted object name
|
|
67
|
+
- Security (CWE-22): confine Visual FoxPro `.dbc` table resolution to the database directory, preventing path traversal via a crafted container object name
|
|
68
|
+
- Security (CWE-1236): neutralize spreadsheet formula injection in `Table#to_csv` by prefixing a quote to string cells/headers starting with `= + - @`
|
|
69
|
+
- Security (CWE-94): escape table and column names when generating ActiveRecord/Sequel schemas, preventing Ruby code injection from crafted DBF header names
|
|
5
70
|
- Add support for 8 more code pages (issue #98): Mazovia cp620 and Kamenický cp895 via vendored translation tables (new DBF::Encoder), plus macRoman, cp1255, cp1256, macCyrillic, macCentEuro and macGreek
|
|
6
71
|
- Blank Visual FoxPro "T" (DateTime) columns now return nil instead of a Julian day-0 date
|
|
7
72
|
- Blank "F" (Float) columns now return nil instead of 0.0, matching "N" (Number) behavior
|
data/LICENSE
CHANGED
data/README.md
CHANGED
|
@@ -2,36 +2,37 @@
|
|
|
2
2
|
|
|
3
3
|
[](https://rubygems.org/gems/dbf)
|
|
4
4
|
[](https://github.com/infused/dbf/actions/workflows/build.yml)
|
|
5
|
-
[](https://codeclimate.com/github/infused/dbf)
|
|
6
|
-
[](https://codeclimate.com/github/infused/dbf)
|
|
7
5
|
[](https://rubygems.org/gems/dbf/)
|
|
8
6
|
[](https://github.com/infused/dbf)
|
|
9
7
|
|
|
10
8
|
DBF is a small, fast Ruby library for reading dBase, xBase, Clipper, and FoxPro database files.
|
|
11
9
|
|
|
12
10
|
* Project page: <https://github.com/infused/dbf>
|
|
13
|
-
* API Documentation: <https://
|
|
11
|
+
* API Documentation: <https://dbf.infused.org>
|
|
14
12
|
* Report bugs: <https://github.com/infused/dbf/issues>
|
|
15
13
|
* Questions: Email <mailto:keithm@infused.org> and put DBF somewhere in the
|
|
16
14
|
subject line
|
|
17
15
|
* Change log: <https://github.com/infused/dbf/blob/main/CHANGELOG.md>
|
|
18
16
|
|
|
19
|
-
NOTE: Beginning with version 5.2 we have dropped support for Ruby 3.2 and earlier.
|
|
20
|
-
|
|
21
|
-
NOTE: Beginning with version 4.3 we have dropped support for Ruby 3.0 and earlier.
|
|
22
|
-
|
|
23
|
-
NOTE: Beginning with version 4 we have dropped support for Ruby 2.0, 2.1, 2.2, and 2.3. If you need support for these older Rubies,
|
|
24
|
-
please use 3.0.x (<https://github.com/infused/dbf/tree/3_stable>)
|
|
25
|
-
|
|
26
|
-
NOTE: Beginning with version 3 we have dropped support for Ruby 1.8 and 1.9. If you need support for older Rubies,
|
|
27
|
-
please use 2.0.x (<https://github.com/infused/dbf/tree/2_stable>)
|
|
28
|
-
|
|
29
17
|
## Compatibility
|
|
30
18
|
|
|
31
19
|
DBF is tested to work with the following versions of Ruby:
|
|
32
20
|
|
|
33
21
|
* Ruby 3.3.x, 3.4.x, 4.0.x
|
|
34
22
|
|
|
23
|
+
Older Rubies are supported by older DBF release lines:
|
|
24
|
+
|
|
25
|
+
| DBF version | Ruby support |
|
|
26
|
+
| ----------- | ------------ |
|
|
27
|
+
| 5.2+ | 3.3+ |
|
|
28
|
+
| 4.3 – 5.1 | 3.1+ |
|
|
29
|
+
| 4.0 – 4.2 | 2.4+ |
|
|
30
|
+
| 3.x ([3_stable](https://github.com/infused/dbf/tree/3_stable)) | 2.0 – 2.3 |
|
|
31
|
+
| 2.x ([2_stable](https://github.com/infused/dbf/tree/2_stable)) | 1.8 – 1.9 |
|
|
32
|
+
|
|
33
|
+
DBF is Ractor-friendly: all library constants are deeply frozen, so tables
|
|
34
|
+
can be opened, enumerated, and exported inside non-main Ractors.
|
|
35
|
+
|
|
35
36
|
## Installation
|
|
36
37
|
|
|
37
38
|
Install the gem manually:
|
|
@@ -58,16 +59,35 @@ widgets = DBF::Table.new("widgets.dbf")
|
|
|
58
59
|
Open a DBF file using an IO object:
|
|
59
60
|
|
|
60
61
|
```ruby
|
|
61
|
-
data = File.open('widgets.dbf')
|
|
62
|
+
data = File.open('widgets.dbf', 'rb')
|
|
62
63
|
widgets = DBF::Table.new(data)
|
|
63
64
|
```
|
|
64
65
|
|
|
66
|
+
All errors raised by the library inherit from DBF::Error, so you can rescue
|
|
67
|
+
the library as a unit:
|
|
68
|
+
|
|
69
|
+
```ruby
|
|
70
|
+
begin
|
|
71
|
+
widgets = DBF::Table.new("widgets.dbf")
|
|
72
|
+
rescue DBF::Error => e
|
|
73
|
+
puts "Unable to read DBF file: #{e.message}"
|
|
74
|
+
end
|
|
75
|
+
```
|
|
76
|
+
|
|
65
77
|
Open a DBF by passing in raw data (wrap the raw data with StringIO):
|
|
66
78
|
|
|
67
79
|
```ruby
|
|
68
80
|
widgets = DBF::Table.new(StringIO.new('raw binary data'))
|
|
69
81
|
```
|
|
70
82
|
|
|
83
|
+
Use the block form to close the table automatically, like File.open:
|
|
84
|
+
|
|
85
|
+
```ruby
|
|
86
|
+
DBF::Table.open('widgets.dbf') do |table|
|
|
87
|
+
table.each { |record| puts record.name }
|
|
88
|
+
end
|
|
89
|
+
```
|
|
90
|
+
|
|
71
91
|
Enumerate all records
|
|
72
92
|
|
|
73
93
|
```ruby
|
|
@@ -287,13 +307,15 @@ table.name = 'my_table_name'
|
|
|
287
307
|
A small command-line utility called dbf is installed with the gem.
|
|
288
308
|
|
|
289
309
|
$ dbf -h
|
|
290
|
-
usage: dbf [-h|-s|-a] filename
|
|
310
|
+
usage: dbf [-h|-s|-a|-c|-r|-j|-J] filename
|
|
291
311
|
-h = print this message
|
|
292
|
-
-v = print the version
|
|
312
|
+
-v = print the DBF gem version
|
|
293
313
|
-s = print summary information
|
|
294
314
|
-a = create an ActiveRecord::Schema
|
|
295
|
-
-r = create a Sequel
|
|
315
|
+
-r = create a Sequel migration
|
|
296
316
|
-c = export as CSV
|
|
317
|
+
-j = export as a JSON array
|
|
318
|
+
-J = export as JSON Lines (one record per line)
|
|
297
319
|
|
|
298
320
|
Create an executable ActiveRecord schema:
|
|
299
321
|
|
|
@@ -307,6 +329,15 @@ Dump all records to a CSV file:
|
|
|
307
329
|
|
|
308
330
|
dbf -c books.dbf > books.csv
|
|
309
331
|
|
|
332
|
+
Dump all records as JSON or JSON Lines:
|
|
333
|
+
|
|
334
|
+
dbf -j books.dbf > books.json
|
|
335
|
+
dbf -J books.dbf > books.jsonl
|
|
336
|
+
|
|
337
|
+
JSON Lines output is streamed record by record, so it works well for very
|
|
338
|
+
large files and pipelines (for example `dbf -J books.dbf | jq` or importing
|
|
339
|
+
into DuckDB).
|
|
340
|
+
|
|
310
341
|
## Reading a Visual Foxpro database (v8, v9)
|
|
311
342
|
|
|
312
343
|
A special Database::Foxpro class is available to read Visual Foxpro container
|
|
@@ -374,25 +405,6 @@ Data type descriptions
|
|
|
374
405
|
|
|
375
406
|
## License
|
|
376
407
|
|
|
377
|
-
Copyright (c) 2006-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
obtaining a copy of this software and associated documentation
|
|
381
|
-
files (the "Software"), to deal in the Software without
|
|
382
|
-
restriction, including without limitation the rights to use,
|
|
383
|
-
copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
384
|
-
copies of the Software, and to permit persons to whom the
|
|
385
|
-
Software is furnished to do so, subject to the following
|
|
386
|
-
conditions:
|
|
387
|
-
|
|
388
|
-
The above copyright notice and this permission notice shall be
|
|
389
|
-
included in all copies or substantial portions of the Software.
|
|
390
|
-
|
|
391
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
|
392
|
-
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
|
|
393
|
-
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
|
394
|
-
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
|
|
395
|
-
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
|
396
|
-
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
|
397
|
-
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
|
|
398
|
-
OTHER DEALINGS IN THE SOFTWARE.
|
|
408
|
+
Copyright (c) 2006-2026 Keith Morrison <<keithm@infused.org>>
|
|
409
|
+
|
|
410
|
+
Released under the MIT License. See [LICENSE](LICENSE) for the full text.
|
data/dbf.gemspec
CHANGED
|
@@ -15,7 +15,11 @@ Gem::Specification.new do |s|
|
|
|
15
15
|
s.require_paths = ['lib']
|
|
16
16
|
s.required_ruby_version = '>= 3.3.0'
|
|
17
17
|
s.metadata['rubygems_mfa_required'] = 'true'
|
|
18
|
+
s.metadata['homepage_uri'] = 'https://github.com/infused/dbf'
|
|
18
19
|
s.metadata['source_code_uri'] = 'https://github.com/infused/dbf'
|
|
19
20
|
s.metadata['changelog_uri'] = 'https://github.com/infused/dbf/blob/main/CHANGELOG.md'
|
|
20
|
-
s.
|
|
21
|
+
s.metadata['bug_tracker_uri'] = 'https://github.com/infused/dbf/issues'
|
|
22
|
+
s.metadata['documentation_uri'] = 'https://dbf.infused.org'
|
|
23
|
+
s.metadata['funding_uri'] = 'https://github.com/sponsors/infused'
|
|
24
|
+
s.add_dependency 'csv', '~> 3.3'
|
|
21
25
|
end
|
data/lib/dbf/cli.rb
CHANGED
|
@@ -5,15 +5,61 @@ require 'optparse'
|
|
|
5
5
|
module DBF
|
|
6
6
|
class CLI
|
|
7
7
|
USAGE = <<~HELP
|
|
8
|
-
usage: dbf [-h|-s|-a|-c|-r] filename
|
|
8
|
+
usage: dbf [-h|-s|-a|-c|-r|-j|-J] filename
|
|
9
9
|
-h = print this message
|
|
10
10
|
-v = print the DBF gem version
|
|
11
11
|
-s = print summary information
|
|
12
12
|
-a = create an ActiveRecord::Schema
|
|
13
13
|
-r = create a Sequel migration
|
|
14
14
|
-c = export as CSV
|
|
15
|
+
-j = export as a JSON array
|
|
16
|
+
-J = export as JSON Lines (one record per line)
|
|
15
17
|
HELP
|
|
16
18
|
|
|
19
|
+
# Bytes a terminal interprets as control or escape sequences. A crafted
|
|
20
|
+
# DBF can carry these in column names and record values, so they are
|
|
21
|
+
# replaced before file-derived text reaches an interactive terminal.
|
|
22
|
+
CONTROL_BYTES = /[\x00-\x1F\x7F]/n
|
|
23
|
+
# The same, but keeping CR and LF so CSV row separators survive.
|
|
24
|
+
CONTROL_BYTES_KEEPING_NEWLINES = /[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]/n
|
|
25
|
+
|
|
26
|
+
# Replaces terminal control bytes. Substitution happens on a byte copy so
|
|
27
|
+
# a value whose bytes are invalid in its encoding cannot raise here.
|
|
28
|
+
#
|
|
29
|
+
# @param value [Object]
|
|
30
|
+
# @param pattern [Regexp]
|
|
31
|
+
# @return [String]
|
|
32
|
+
def self.sanitize(value, pattern = CONTROL_BYTES)
|
|
33
|
+
string = value.to_s
|
|
34
|
+
string.b.gsub(pattern, '?').force_encoding(string.encoding)
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
# Wraps an IO so text written to an interactive terminal is stripped of
|
|
38
|
+
# control bytes. Redirected or piped output is never wrapped, so exported
|
|
39
|
+
# data is passed through unaltered.
|
|
40
|
+
class TerminalFilter
|
|
41
|
+
def initialize(io)
|
|
42
|
+
@io = io
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
def <<(data)
|
|
46
|
+
@io << CLI.sanitize(data, CONTROL_BYTES_KEEPING_NEWLINES)
|
|
47
|
+
self
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def write(*data)
|
|
51
|
+
@io.write(*data.map { |datum| CLI.sanitize(datum, CONTROL_BYTES_KEEPING_NEWLINES) })
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
def method_missing(name, ...) # :nodoc:
|
|
55
|
+
@io.respond_to?(name) ? @io.send(name, ...) : super
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
def respond_to_missing?(name, include_private = false) # :nodoc:
|
|
59
|
+
@io.respond_to?(name, include_private) || super
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
|
|
17
63
|
def self.run(argv, stdout: $stdout, stderr: $stderr)
|
|
18
64
|
new(argv, stdout: stdout, stderr: stderr).run
|
|
19
65
|
end
|
|
@@ -24,8 +70,17 @@ module DBF
|
|
|
24
70
|
@stderr = stderr
|
|
25
71
|
end
|
|
26
72
|
|
|
73
|
+
ACTIONS = {
|
|
74
|
+
'a' => :print_ar_schema,
|
|
75
|
+
'r' => :print_sequel_schema,
|
|
76
|
+
's' => :print_summary,
|
|
77
|
+
'c' => :print_csv,
|
|
78
|
+
'j' => :print_json,
|
|
79
|
+
'J' => :print_jsonl
|
|
80
|
+
}.freeze
|
|
81
|
+
|
|
27
82
|
def run
|
|
28
|
-
params = OptionParser.new.getopts(@argv, '
|
|
83
|
+
params = OptionParser.new.getopts(@argv, 'hsacrvjJ')
|
|
29
84
|
|
|
30
85
|
if params['v']
|
|
31
86
|
print_version
|
|
@@ -35,13 +90,8 @@ module DBF
|
|
|
35
90
|
filename = @argv.shift
|
|
36
91
|
return missing_filename unless filename
|
|
37
92
|
|
|
38
|
-
action =
|
|
39
|
-
|
|
40
|
-
when 'a' then print_ar_schema(filename)
|
|
41
|
-
when 'r' then print_sequel_schema(filename)
|
|
42
|
-
when 's' then print_summary(filename)
|
|
43
|
-
when 'c' then print_csv(filename)
|
|
44
|
-
end
|
|
93
|
+
action = ACTIONS.find { |flag, _method| params[flag] }&.last
|
|
94
|
+
send(action, filename) if action
|
|
45
95
|
end
|
|
46
96
|
0
|
|
47
97
|
rescue DBF::FileNotFoundError => e
|
|
@@ -64,16 +114,23 @@ module DBF
|
|
|
64
114
|
1
|
|
65
115
|
end
|
|
66
116
|
|
|
117
|
+
# Always close the table when done: an open handle prevents deleting or
|
|
118
|
+
# replacing the file on Windows.
|
|
119
|
+
def with_table(filename, &) = DBF::Table.open(filename, &)
|
|
120
|
+
|
|
67
121
|
def print_ar_schema(filename)
|
|
68
|
-
@stdout.puts
|
|
122
|
+
with_table(filename) { |table| @stdout.puts terminal_safe(table.schema(:activerecord)) }
|
|
69
123
|
end
|
|
70
124
|
|
|
71
125
|
def print_sequel_schema(filename)
|
|
72
|
-
@stdout.puts
|
|
126
|
+
with_table(filename) { |table| @stdout.puts terminal_safe(table.schema(:sequel)) }
|
|
73
127
|
end
|
|
74
128
|
|
|
75
129
|
def print_summary(filename)
|
|
76
|
-
table
|
|
130
|
+
with_table(filename) { |table| write_summary(filename, table) }
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
def write_summary(filename, table)
|
|
77
134
|
@stdout.puts
|
|
78
135
|
@stdout.puts "Database: #{filename}"
|
|
79
136
|
@stdout.puts "Type: (#{table.version}) #{table.version_description}"
|
|
@@ -84,12 +141,74 @@ module DBF
|
|
|
84
141
|
@stdout.puts 'Name Type Length Decimal'
|
|
85
142
|
@stdout.puts '-' * 78
|
|
86
143
|
table.columns.each do |f|
|
|
87
|
-
|
|
144
|
+
# Column names and types come from the file. Always replace control
|
|
145
|
+
# bytes here: they are never valid in a name and would otherwise both
|
|
146
|
+
# emit escape sequences and break the column alignment below.
|
|
147
|
+
@stdout.puts format('%-16s %-10s %-10s %-10s', self.class.sanitize(f.name), self.class.sanitize(f.type), f.length, f.decimal)
|
|
88
148
|
end
|
|
89
149
|
end
|
|
90
150
|
|
|
91
151
|
def print_csv(filename)
|
|
92
|
-
|
|
152
|
+
with_table(filename) { |table| table.to_csv(interactive? ? TerminalFilter.new(@stdout) : @stdout) }
|
|
153
|
+
end
|
|
154
|
+
|
|
155
|
+
# Streams a JSON array without materializing all records in memory.
|
|
156
|
+
# JSON string escaping makes the output terminal-safe by construction:
|
|
157
|
+
# control bytes are emitted as \uXXXX escapes.
|
|
158
|
+
def print_json(filename)
|
|
159
|
+
with_table(filename) { |table| write_json(table) }
|
|
160
|
+
end
|
|
161
|
+
|
|
162
|
+
def write_json(table)
|
|
163
|
+
first = true
|
|
164
|
+
@stdout.write('[')
|
|
165
|
+
each_present_record(table) do |record|
|
|
166
|
+
@stdout.write(first ? "\n" : ",\n")
|
|
167
|
+
@stdout.write(json_record(record))
|
|
168
|
+
first = false
|
|
169
|
+
end
|
|
170
|
+
@stdout.write("\n]\n")
|
|
171
|
+
end
|
|
172
|
+
|
|
173
|
+
# One JSON object per line (JSON Lines). Combined with chunked record
|
|
174
|
+
# reading this exports arbitrarily large files in constant memory.
|
|
175
|
+
def print_jsonl(filename)
|
|
176
|
+
with_table(filename) { |table| write_jsonl(table) }
|
|
177
|
+
end
|
|
178
|
+
|
|
179
|
+
def write_jsonl(table)
|
|
180
|
+
each_present_record(table) { |record| @stdout.write("#{json_record(record)}\n") }
|
|
181
|
+
end
|
|
182
|
+
|
|
183
|
+
# Deleted records have no attributes, so JSON export skips them.
|
|
184
|
+
def each_present_record(table, &)
|
|
185
|
+
table.each { |record| yield record if record }
|
|
186
|
+
end
|
|
187
|
+
|
|
188
|
+
def json_record(record)
|
|
189
|
+
JSON.generate(record.attributes.to_h { |key, value| [json_safe(key), json_safe(value)] })
|
|
190
|
+
end
|
|
191
|
+
|
|
192
|
+
# JSON.generate raises on binary or invalidly encoded strings; represent
|
|
193
|
+
# their bytes instead of raising, mirroring the CSV export behavior.
|
|
194
|
+
def json_safe(value)
|
|
195
|
+
return value unless value.is_a?(::String)
|
|
196
|
+
|
|
197
|
+
value.encode(Encoding::UTF_8, invalid: :replace, undef: :replace, replace: '?')
|
|
198
|
+
rescue Encoding::ConverterNotFoundError
|
|
199
|
+
value.dup.force_encoding(Encoding::UTF_8).scrub('?')
|
|
200
|
+
end
|
|
201
|
+
|
|
202
|
+
# Exported data is only filtered when it is going to a terminal, so
|
|
203
|
+
# redirecting or piping still produces byte-for-byte the original values.
|
|
204
|
+
def terminal_safe(text)
|
|
205
|
+
return text unless interactive?
|
|
206
|
+
|
|
207
|
+
self.class.sanitize(text, CONTROL_BYTES_KEEPING_NEWLINES)
|
|
208
|
+
end
|
|
209
|
+
|
|
210
|
+
def interactive?
|
|
211
|
+
@stdout.respond_to?(:tty?) && @stdout.tty?
|
|
93
212
|
end
|
|
94
213
|
end
|
|
95
214
|
end
|
data/lib/dbf/column.rb
CHANGED
|
@@ -2,12 +2,17 @@
|
|
|
2
2
|
|
|
3
3
|
module DBF
|
|
4
4
|
class Column
|
|
5
|
-
class LengthError <
|
|
5
|
+
class LengthError < DBF::Error
|
|
6
6
|
end
|
|
7
7
|
|
|
8
|
-
class
|
|
8
|
+
class InvalidNameError < DBF::Error
|
|
9
9
|
end
|
|
10
10
|
|
|
11
|
+
# Deprecated alias, kept for backward compatibility. It shadowed
|
|
12
|
+
# ::NameError (without being one), so it was renamed.
|
|
13
|
+
NameError = InvalidNameError
|
|
14
|
+
deprecate_constant :NameError
|
|
15
|
+
|
|
11
16
|
attr_reader :name, :type, :length, :decimal
|
|
12
17
|
|
|
13
18
|
# rubocop:disable Style/MutableConstant
|
|
@@ -38,7 +43,7 @@ module DBF
|
|
|
38
43
|
def initialize(table, name, type, length, decimal)
|
|
39
44
|
@table = table
|
|
40
45
|
@name = clean(name)
|
|
41
|
-
@type = type
|
|
46
|
+
@type = clean_type(type)
|
|
42
47
|
@length = length
|
|
43
48
|
@decimal = decimal
|
|
44
49
|
|
|
@@ -85,6 +90,14 @@ module DBF
|
|
|
85
90
|
@table.encode_string(value.strip.split("\x00", 2).first || +'')
|
|
86
91
|
end
|
|
87
92
|
|
|
93
|
+
# The column type is a single ASCII character. A corrupt file can supply
|
|
94
|
+
# any byte, which would otherwise stay binary and raise when serialized
|
|
95
|
+
# (e.g. to JSON), so replace anything unrepresentable.
|
|
96
|
+
def clean_type(value) # :nodoc:
|
|
97
|
+
type = value.to_s.dup.force_encoding(Encoding::UTF_8)
|
|
98
|
+
type.valid_encoding? ? type : type.scrub('?')
|
|
99
|
+
end
|
|
100
|
+
|
|
88
101
|
def type_cast_class # :nodoc:
|
|
89
102
|
@type_cast_class ||= begin
|
|
90
103
|
klass = @length == 0 ? ColumnType::Nil : TYPE_CAST_CLASS[type.to_sym]
|
|
@@ -97,7 +110,7 @@ module DBF
|
|
|
97
110
|
end
|
|
98
111
|
|
|
99
112
|
def validate_name # :nodoc:
|
|
100
|
-
raise
|
|
113
|
+
raise InvalidNameError, 'column name cannot be empty' if @name.empty?
|
|
101
114
|
end
|
|
102
115
|
end
|
|
103
116
|
end
|
data/lib/dbf/column_builder.rb
CHANGED
|
@@ -12,7 +12,15 @@ module DBF
|
|
|
12
12
|
safe_seek do
|
|
13
13
|
@data.seek(@version_config.header_size)
|
|
14
14
|
[].tap do |columns|
|
|
15
|
-
|
|
15
|
+
until end_of_record?
|
|
16
|
+
args = @version_config.read_column_args(@table, @data)
|
|
17
|
+
# A descriptor truncated by EOF is returned as nil, or unpacks to
|
|
18
|
+
# a nil length; stop rather than constructing an invalid column
|
|
19
|
+
# (which would crash on nil < 0).
|
|
20
|
+
break if args.nil? || args[3].nil?
|
|
21
|
+
|
|
22
|
+
columns << Column.new(*args)
|
|
23
|
+
end
|
|
16
24
|
end
|
|
17
25
|
end
|
|
18
26
|
end
|
|
@@ -20,7 +28,12 @@ module DBF
|
|
|
20
28
|
private
|
|
21
29
|
|
|
22
30
|
def end_of_record?
|
|
23
|
-
safe_seek
|
|
31
|
+
safe_seek do
|
|
32
|
+
byte = @data.read(1)
|
|
33
|
+
# A truncated file that ends before the 0x0D column terminator marks
|
|
34
|
+
# the end of the column list rather than crashing on nil.ord.
|
|
35
|
+
byte.nil? || byte.ord == 13
|
|
36
|
+
end
|
|
24
37
|
end
|
|
25
38
|
|
|
26
39
|
def safe_seek
|
data/lib/dbf/column_type.rb
CHANGED
|
@@ -20,6 +20,10 @@ module DBF
|
|
|
20
20
|
end
|
|
21
21
|
|
|
22
22
|
def decode(raw, &)
|
|
23
|
+
# A record truncated before this column yields a nil slice; treat it
|
|
24
|
+
# as blank rather than crashing in the type cast.
|
|
25
|
+
return blank_value if raw.nil?
|
|
26
|
+
|
|
23
27
|
if skip_blank? && raw.count(' ') == raw.length
|
|
24
28
|
blank_value
|
|
25
29
|
else
|
|
@@ -49,7 +53,8 @@ module DBF
|
|
|
49
53
|
class Currency < Base
|
|
50
54
|
# @param value [String]
|
|
51
55
|
def type_cast(value)
|
|
52
|
-
|
|
56
|
+
int = value.unpack1('q<')
|
|
57
|
+
int && (int / 10_000.0).to_f
|
|
53
58
|
end
|
|
54
59
|
end
|
|
55
60
|
|
|
@@ -64,6 +69,8 @@ module DBF
|
|
|
64
69
|
# @param value [String]
|
|
65
70
|
def type_cast(value)
|
|
66
71
|
bits = value.unpack1('B*')
|
|
72
|
+
return nil unless bits && bits.length >= 32
|
|
73
|
+
|
|
67
74
|
sign_multiplier = bits[0] == '0' ? -1 : 1
|
|
68
75
|
bits[1, 31].to_i(2) * sign_multiplier
|
|
69
76
|
end
|
|
@@ -105,7 +112,7 @@ module DBF
|
|
|
105
112
|
# @param value [String]
|
|
106
113
|
def type_cast(value)
|
|
107
114
|
value.match?(/\d{8}/) && ::Date.strptime(value, '%Y%m%d')
|
|
108
|
-
rescue
|
|
115
|
+
rescue ArgumentError, RangeError, TypeError
|
|
109
116
|
nil
|
|
110
117
|
end
|
|
111
118
|
end
|
|
@@ -118,7 +125,7 @@ module DBF
|
|
|
118
125
|
|
|
119
126
|
secs = (msecs / 1000).to_i
|
|
120
127
|
::DateTime.jd(days, (secs / 3600).to_i, (secs / 60).to_i % 60, secs % 60).to_time
|
|
121
|
-
rescue
|
|
128
|
+
rescue ArgumentError, RangeError, TypeError
|
|
122
129
|
nil
|
|
123
130
|
end
|
|
124
131
|
end
|
data/lib/dbf/database/foxpro.rb
CHANGED
|
@@ -47,11 +47,25 @@ module DBF
|
|
|
47
47
|
# @param name [String]
|
|
48
48
|
# @return [String]
|
|
49
49
|
def table_path(name)
|
|
50
|
-
|
|
51
|
-
|
|
50
|
+
name = name.to_s
|
|
51
|
+
raise DBF::FileNotFoundError, "related table not found: #{name}" if name.empty? || name.include?("\x00")
|
|
52
|
+
|
|
53
|
+
# Treat the container-supplied name as an untrusted basename so that
|
|
54
|
+
# path separators and ".." cannot escape the database directory.
|
|
55
|
+
# Match case-insensitively by scanning the directory rather than
|
|
56
|
+
# globbing, so glob metacharacters (`{}`, `*`, `?`, `[]`) in the name
|
|
57
|
+
# cannot trigger brace-expansion CPU exhaustion.
|
|
58
|
+
target = "#{File.basename(name)}.dbf"
|
|
59
|
+
entry = Dir.children(@dirname).find { |child| child.casecmp?(target) }
|
|
60
|
+
path = entry && File.join(@dirname, entry)
|
|
52
61
|
|
|
53
62
|
raise DBF::FileNotFoundError, "related table not found: #{name}" unless path && File.exist?(path)
|
|
54
63
|
|
|
64
|
+
# Defense in depth: confirm the resolved file really is inside the
|
|
65
|
+
# database directory before opening it.
|
|
66
|
+
contained = File.realpath(path).start_with?("#{File.realpath(@dirname)}#{File::SEPARATOR}")
|
|
67
|
+
raise DBF::FileNotFoundError, "related table not found: #{name}" unless contained
|
|
68
|
+
|
|
55
69
|
path
|
|
56
70
|
end
|
|
57
71
|
|
|
@@ -100,12 +114,10 @@ module DBF
|
|
|
100
114
|
end
|
|
101
115
|
|
|
102
116
|
def build_columns # :nodoc:
|
|
103
|
-
columns = super
|
|
104
|
-
|
|
105
117
|
# modify the column definitions to use the long names as the
|
|
106
118
|
# columnname property is readonly, recreate the column definitions
|
|
107
|
-
|
|
108
|
-
long_name = long_names[
|
|
119
|
+
super.each_with_index.map do |column, index|
|
|
120
|
+
long_name = long_names[index] || column.name
|
|
109
121
|
Column.new(self, long_name, column.type, column.length, column.decimal)
|
|
110
122
|
end
|
|
111
123
|
end
|
data/lib/dbf/errors.rb
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module DBF
|
|
4
|
+
# Base class for all errors raised by the DBF library, so callers can
|
|
5
|
+
# rescue DBF::Error to catch anything the library raises.
|
|
6
|
+
class Error < StandardError
|
|
7
|
+
end
|
|
8
|
+
|
|
9
|
+
class FileNotFoundError < Error
|
|
10
|
+
end
|
|
11
|
+
|
|
12
|
+
class NoColumnsDefined < Error
|
|
13
|
+
end
|
|
14
|
+
end
|
data/lib/dbf/file_handler.rb
CHANGED
|
@@ -11,7 +11,12 @@ module DBF
|
|
|
11
11
|
when String
|
|
12
12
|
File.open(data, 'rb')
|
|
13
13
|
else
|
|
14
|
-
raise ArgumentError, 'data must be a file path or
|
|
14
|
+
raise ArgumentError, 'data must be a file path or an IO-like object responding to #read and #seek' unless data.respond_to?(:read) && data.respond_to?(:seek)
|
|
15
|
+
|
|
16
|
+
# DBF is a binary format; a File opened in text mode would corrupt
|
|
17
|
+
# reads on Windows.
|
|
18
|
+
data.binmode if data.respond_to?(:binmode)
|
|
19
|
+
data
|
|
15
20
|
end
|
|
16
21
|
rescue Errno::ENOENT
|
|
17
22
|
raise DBF::FileNotFoundError, "file not found: #{data}"
|
|
@@ -19,17 +24,23 @@ module DBF
|
|
|
19
24
|
|
|
20
25
|
def open_memo(data, memo, memo_class, version)
|
|
21
26
|
if memo
|
|
22
|
-
meth = memo.is_a?(
|
|
27
|
+
meth = memo.is_a?(String) ? :open : :new
|
|
23
28
|
memo_class.send(meth, memo, version)
|
|
24
|
-
elsif
|
|
25
|
-
|
|
26
|
-
|
|
29
|
+
elsif (path = data_path(data))
|
|
30
|
+
found = Dir.glob(memo_search_path(path)).first
|
|
31
|
+
found && memo_class.open(found, version)
|
|
27
32
|
end
|
|
28
33
|
end
|
|
29
34
|
|
|
30
|
-
def
|
|
31
|
-
|
|
32
|
-
|
|
35
|
+
def data_path(data)
|
|
36
|
+
return data if data.is_a?(String)
|
|
37
|
+
|
|
38
|
+
data.path if data.respond_to?(:path)
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def memo_search_path(path)
|
|
42
|
+
dirname = File.dirname(path)
|
|
43
|
+
basename = File.basename(path, '.*')
|
|
33
44
|
"#{dirname}/#{basename}*.{fpt,FPT,dbt,DBT}"
|
|
34
45
|
end
|
|
35
46
|
end
|
data/lib/dbf/header.rb
CHANGED
|
@@ -5,6 +5,9 @@ module DBF
|
|
|
5
5
|
attr_reader :version, :record_count, :header_length, :record_length, :encoding_key, :encoding
|
|
6
6
|
|
|
7
7
|
def initialize(data)
|
|
8
|
+
# Pad a nil or truncated header read so unpacking a short file yields
|
|
9
|
+
# empty values instead of raising.
|
|
10
|
+
data = data.to_s.b.ljust(VersionConfig::DBASE3_HEADER_SIZE, "\x00")
|
|
8
11
|
@version = data.unpack1('H2')
|
|
9
12
|
@encoding_key = nil
|
|
10
13
|
@encoding = nil
|
data/lib/dbf/memo/dbase3.rb
CHANGED
|
@@ -7,7 +7,12 @@ module DBF
|
|
|
7
7
|
data.seek offset(start_block)
|
|
8
8
|
memo_string = +''
|
|
9
9
|
loop do
|
|
10
|
-
block = data.read(BLOCK_SIZE)
|
|
10
|
+
block = data.read(BLOCK_SIZE)
|
|
11
|
+
# A start block past EOF yields nil; return what we have rather
|
|
12
|
+
# than crashing on nil.gsub.
|
|
13
|
+
break if block.nil?
|
|
14
|
+
|
|
15
|
+
block = block.gsub(/(\000|\032)/, '')
|
|
11
16
|
memo_string << block
|
|
12
17
|
break if block.size < BLOCK_SIZE
|
|
13
18
|
end
|
data/lib/dbf/memo/dbase4.rb
CHANGED
|
@@ -5,7 +5,21 @@ module DBF
|
|
|
5
5
|
class Dbase4 < Base
|
|
6
6
|
def build_memo(start_block) # :nodoc:
|
|
7
7
|
data.seek offset(start_block)
|
|
8
|
-
|
|
8
|
+
|
|
9
|
+
# A start block past EOF yields a nil/short header; return nil rather
|
|
10
|
+
# than crashing on nil.unpack1.
|
|
11
|
+
header = data.read(BLOCK_HEADER_SIZE)
|
|
12
|
+
return nil unless header && header.bytesize == BLOCK_HEADER_SIZE
|
|
13
|
+
|
|
14
|
+
length = header.unpack1('x4L')
|
|
15
|
+
|
|
16
|
+
# Bound the read by the bytes remaining so a crafted 32-bit length
|
|
17
|
+
# field cannot force a ~4 GiB allocation from a small memo file.
|
|
18
|
+
remaining = data.size - data.pos
|
|
19
|
+
length = remaining if length > remaining
|
|
20
|
+
return nil if length <= 0
|
|
21
|
+
|
|
22
|
+
data.read(length)
|
|
9
23
|
end
|
|
10
24
|
end
|
|
11
25
|
end
|
data/lib/dbf/memo/foxpro.rb
CHANGED
|
@@ -5,35 +5,40 @@ module DBF
|
|
|
5
5
|
class Foxpro < Base
|
|
6
6
|
FPT_HEADER_SIZE = 512
|
|
7
7
|
|
|
8
|
-
def initialize(data, version)
|
|
9
|
-
@data = data
|
|
10
|
-
super
|
|
11
|
-
end
|
|
12
|
-
|
|
13
8
|
def build_memo(start_block) # :nodoc:
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
return nil unless
|
|
9
|
+
data.seek offset(start_block)
|
|
10
|
+
block = data.read(block_size)
|
|
11
|
+
return nil unless block
|
|
12
|
+
|
|
13
|
+
# memo_size is nil when the block header is truncated (< 8 bytes)
|
|
14
|
+
memo_type, memo_size, memo_string = block.unpack('NNa*')
|
|
15
|
+
return nil unless memo_type == 1 && memo_size.to_i.positive?
|
|
17
16
|
|
|
18
17
|
read_memo_content(memo_string, memo_size)
|
|
19
|
-
rescue
|
|
18
|
+
rescue IOError, SystemCallError, RangeError
|
|
20
19
|
nil
|
|
21
20
|
end
|
|
22
21
|
|
|
23
22
|
private
|
|
24
23
|
|
|
25
24
|
def read_memo_content(memo_string, memo_size) # :nodoc:
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
25
|
+
return memo_string[0, memo_size] unless memo_size > block_content_size
|
|
26
|
+
|
|
27
|
+
# Bound the read by the bytes remaining so a crafted 32-bit memo_size
|
|
28
|
+
# cannot force a ~4 GiB allocation from a small memo file.
|
|
29
|
+
length = content_size(memo_size)
|
|
30
|
+
remaining = data.size - data.pos
|
|
31
|
+
length = remaining if length > remaining
|
|
32
|
+
memo_string << data.read(length) if length.positive?
|
|
33
|
+
memo_string
|
|
31
34
|
end
|
|
32
35
|
|
|
33
36
|
def block_size # :nodoc:
|
|
34
37
|
@block_size ||= begin
|
|
35
|
-
|
|
36
|
-
|
|
38
|
+
data.rewind
|
|
39
|
+
header = data.read(FPT_HEADER_SIZE)
|
|
40
|
+
# A header shorter than 8 bytes cannot contain the block size field
|
|
41
|
+
header && header.bytesize >= 8 ? header.unpack1('x6n') : 0
|
|
37
42
|
end
|
|
38
43
|
end
|
|
39
44
|
end
|
data/lib/dbf/record.rb
CHANGED
|
@@ -20,9 +20,7 @@ module DBF
|
|
|
20
20
|
# @param [DBF::Record] other
|
|
21
21
|
# @return [Boolean]
|
|
22
22
|
def ==(other)
|
|
23
|
-
attributes == other.attributes
|
|
24
|
-
rescue NoMethodError
|
|
25
|
-
false
|
|
23
|
+
other.respond_to?(:attributes) && attributes == other.attributes
|
|
26
24
|
end
|
|
27
25
|
|
|
28
26
|
# Reads attributes by column name
|
|
@@ -83,7 +81,9 @@ module DBF
|
|
|
83
81
|
|
|
84
82
|
def decode_memo_value(raw) # :nodoc:
|
|
85
83
|
memo = @context.memo
|
|
86
|
-
|
|
84
|
+
# A record truncated before the memo column yields a nil pointer; skip
|
|
85
|
+
# decoding rather than crashing on nil.unpack1.
|
|
86
|
+
return nil unless memo && raw
|
|
87
87
|
|
|
88
88
|
version = @context.version
|
|
89
89
|
raw = raw.unpack1('V') if version == '30' || version == '31'
|
data/lib/dbf/record_context.rb
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
module DBF
|
|
4
|
-
|
|
4
|
+
# Immutable per-table state shared by every record during decoding
|
|
5
|
+
RecordContext = Data.define(:columns, :version, :memo, :column_offsets)
|
|
5
6
|
end
|
data/lib/dbf/record_iterator.rb
CHANGED
|
@@ -2,34 +2,75 @@
|
|
|
2
2
|
|
|
3
3
|
module DBF
|
|
4
4
|
class RecordIterator
|
|
5
|
-
|
|
5
|
+
# Records are read in chunks of whole records totalling roughly this many
|
|
6
|
+
# bytes, so enumerating a multi-gigabyte file (for example a shapefile
|
|
7
|
+
# sidecar) needs only chunk-sized memory instead of the entire record
|
|
8
|
+
# section at once.
|
|
9
|
+
CHUNK_SIZE = 4 * 1024 * 1024
|
|
10
|
+
|
|
11
|
+
def initialize(data, context, header_length, record_length, record_count, chunk_size: CHUNK_SIZE)
|
|
6
12
|
@data = data
|
|
7
13
|
@context = context
|
|
8
14
|
@header_length = header_length
|
|
9
15
|
@record_length = record_length
|
|
10
16
|
@record_count = record_count
|
|
17
|
+
@chunk_size = chunk_size
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def each(&)
|
|
21
|
+
return enum_for(:each) unless block_given?
|
|
22
|
+
|
|
23
|
+
# A record_length of 0 from a crafted header cannot drive an unbounded
|
|
24
|
+
# loop: capacity is 0 and enumeration ends immediately.
|
|
25
|
+
remaining = record_capacity
|
|
26
|
+
@data.seek(@header_length)
|
|
27
|
+
|
|
28
|
+
while remaining.positive?
|
|
29
|
+
wanted = [per_chunk, remaining].min
|
|
30
|
+
buffer = @data.read(wanted * @record_length)
|
|
31
|
+
break unless buffer
|
|
32
|
+
|
|
33
|
+
whole_records = buffer.bytesize / @record_length
|
|
34
|
+
break if whole_records.zero?
|
|
35
|
+
|
|
36
|
+
yield_chunk(buffer, whole_records, &)
|
|
37
|
+
remaining -= whole_records
|
|
38
|
+
|
|
39
|
+
# A short read means the file ended earlier than the header promised
|
|
40
|
+
break if whole_records < wanted
|
|
41
|
+
end
|
|
11
42
|
end
|
|
12
43
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
44
|
+
private
|
|
45
|
+
|
|
46
|
+
# Whole records per read; at least one so a record larger than the chunk
|
|
47
|
+
# size still makes progress. Only called when record_length is positive.
|
|
48
|
+
def per_chunk
|
|
49
|
+
@per_chunk ||= [@chunk_size / @record_length, 1].max
|
|
50
|
+
end
|
|
16
51
|
|
|
52
|
+
def yield_chunk(buffer, count)
|
|
17
53
|
pos = 0
|
|
18
|
-
|
|
19
|
-
if
|
|
54
|
+
count.times do
|
|
55
|
+
if buffer.getbyte(pos) == 0x2A
|
|
20
56
|
yield nil
|
|
21
57
|
else
|
|
22
|
-
yield Record.new(
|
|
58
|
+
yield Record.new(buffer, @context, pos + 1)
|
|
23
59
|
end
|
|
24
60
|
pos += @record_length
|
|
25
61
|
end
|
|
26
62
|
end
|
|
27
63
|
|
|
28
|
-
|
|
64
|
+
# Bound enumeration by the bytes actually available so a crafted header
|
|
65
|
+
# (huge record_length * record_count) cannot force reads past the real
|
|
66
|
+
# file size, while record_count still caps a file with trailing garbage.
|
|
67
|
+
def record_capacity
|
|
68
|
+
return 0 unless @record_length.positive?
|
|
29
69
|
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
70
|
+
available = @data.size - @header_length
|
|
71
|
+
return 0 if available.negative?
|
|
72
|
+
|
|
73
|
+
[@record_count, available / @record_length].min
|
|
33
74
|
end
|
|
34
75
|
end
|
|
35
76
|
end
|
data/lib/dbf/schema.rb
CHANGED
|
@@ -43,10 +43,9 @@ module DBF
|
|
|
43
43
|
# @param table_only [Boolean]
|
|
44
44
|
# @return [String]
|
|
45
45
|
def schema(format = :activerecord, table_only: false)
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
raise ArgumentError, ":#{format} is not a valid schema. Valid schemas are: #{FORMATS.join(', ')}."
|
|
46
|
+
raise ArgumentError, ":#{format} is not a valid schema. Valid schemas are: #{FORMATS.join(', ')}." unless FORMATS.include?(format.to_s.to_sym)
|
|
47
|
+
|
|
48
|
+
send(schema_name(format), table_only: table_only)
|
|
50
49
|
end
|
|
51
50
|
|
|
52
51
|
def schema_name(format) # :nodoc:
|
|
@@ -55,7 +54,7 @@ module DBF
|
|
|
55
54
|
|
|
56
55
|
def activerecord_schema(*) # :nodoc:
|
|
57
56
|
output = +"ActiveRecord::Schema.define do\n"
|
|
58
|
-
output << " create_table
|
|
57
|
+
output << " create_table #{name.to_s.inspect} do |t|\n"
|
|
59
58
|
columns.each do |column|
|
|
60
59
|
output << " t.column #{activerecord_schema_definition(column)}"
|
|
61
60
|
end
|
|
@@ -66,7 +65,7 @@ module DBF
|
|
|
66
65
|
def sequel_schema(table_only: false) # :nodoc:
|
|
67
66
|
output = +''
|
|
68
67
|
output << "Sequel.migration do\n change do\n " unless table_only
|
|
69
|
-
output << " create_table(
|
|
68
|
+
output << " create_table(#{name.to_s.to_sym.inspect}) do\n"
|
|
70
69
|
columns.each do |column|
|
|
71
70
|
output << " column #{sequel_schema_definition(column)}"
|
|
72
71
|
end
|
|
@@ -84,7 +83,7 @@ module DBF
|
|
|
84
83
|
# @param column [DBF::Column]
|
|
85
84
|
# @return [String]
|
|
86
85
|
def activerecord_schema_definition(column)
|
|
87
|
-
"
|
|
86
|
+
"#{column.underscored_name.inspect}, #{schema_data_type(column, :activerecord)}\n"
|
|
88
87
|
end
|
|
89
88
|
|
|
90
89
|
# Sequel schema definition
|
|
@@ -92,7 +91,7 @@ module DBF
|
|
|
92
91
|
# @param column [DBF::Column]
|
|
93
92
|
# @return [String]
|
|
94
93
|
def sequel_schema_definition(column)
|
|
95
|
-
"
|
|
94
|
+
"#{column.underscored_name.to_sym.inspect}, #{schema_data_type(column, :sequel)}\n"
|
|
96
95
|
end
|
|
97
96
|
|
|
98
97
|
def schema_data_type(column, format = :activerecord) # :nodoc:
|
data/lib/dbf/table.rb
CHANGED
|
@@ -1,11 +1,9 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
module DBF
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
class NoColumnsDefined < StandardError
|
|
8
|
-
end
|
|
4
|
+
# Leading bytes that make a spreadsheet treat a CSV cell as a formula:
|
|
5
|
+
# "=", "+", "-", "@", tab, and carriage return.
|
|
6
|
+
CSV_FORMULA_TRIGGERS = [0x3D, 0x2B, 0x2D, 0x40, 0x09, 0x0D].freeze
|
|
9
7
|
|
|
10
8
|
# DBF::Table is the primary interface to a single DBF file and provides
|
|
11
9
|
# methods for enumerating and searching the records.
|
|
@@ -33,6 +31,9 @@ module DBF
|
|
|
33
31
|
# # working with a dbf in memory
|
|
34
32
|
# table = DBF::Table.new StringIO.new(dbf_data)
|
|
35
33
|
#
|
|
34
|
+
# # working with an open IO object
|
|
35
|
+
# table = DBF::Table.new File.open('data.dbf', 'rb')
|
|
36
|
+
#
|
|
36
37
|
# # working with a dbf and memo in memory
|
|
37
38
|
# table = DBF::Table.new StringIO.new(dbf_data), StringIO.new(memo_data)
|
|
38
39
|
#
|
|
@@ -40,8 +41,28 @@ module DBF
|
|
|
40
41
|
# table = DBF::Table.new 'data.dbf', nil, 'cp437'
|
|
41
42
|
# table = DBF::Table.new 'data.dbf', 'memo.dbt', Encoding::US_ASCII
|
|
42
43
|
#
|
|
43
|
-
#
|
|
44
|
-
#
|
|
44
|
+
# Opens a table like .new, but when given a block, yields the table,
|
|
45
|
+
# closes it when the block returns, and returns the block's value —
|
|
46
|
+
# the same contract as File.open.
|
|
47
|
+
#
|
|
48
|
+
# DBF::Table.open('data.dbf') do |table|
|
|
49
|
+
# table.each { |record| ... }
|
|
50
|
+
# end
|
|
51
|
+
#
|
|
52
|
+
# Takes the same arguments as .new. Without a block, equivalent to .new.
|
|
53
|
+
def self.open(data, memo = nil, encoding = nil, name: nil)
|
|
54
|
+
table = new(data, memo, encoding, name: name)
|
|
55
|
+
return table unless block_given?
|
|
56
|
+
|
|
57
|
+
begin
|
|
58
|
+
yield table
|
|
59
|
+
ensure
|
|
60
|
+
table.close
|
|
61
|
+
end
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
# @param data [String, StringIO, IO] data Path to the dbf file or an IO-like object
|
|
65
|
+
# @param memo [optional String, StringIO, IO] memo Path to the memo file or an IO-like object
|
|
45
66
|
# @param encoding [optional String, Encoding] encoding Name of the encoding or an Encoding object
|
|
46
67
|
def initialize(data, memo = nil, encoding = nil, name: nil)
|
|
47
68
|
@data = FileHandler.open_data(data)
|
|
@@ -106,7 +127,7 @@ module DBF
|
|
|
106
127
|
|
|
107
128
|
# @return [String]
|
|
108
129
|
def filename
|
|
109
|
-
File.basename(@data.path) if @data.
|
|
130
|
+
File.basename(@data.path) if @data.respond_to?(:path)
|
|
110
131
|
end
|
|
111
132
|
|
|
112
133
|
# @return [TrueClass, FalseClass]
|
|
@@ -132,6 +153,10 @@ module DBF
|
|
|
132
153
|
return nil if deleted_record?
|
|
133
154
|
|
|
134
155
|
record_data = @data.read(record_length)
|
|
156
|
+
# A file that ends immediately after the delete flag has no record body;
|
|
157
|
+
# treat it as absent rather than building a Record over nil data.
|
|
158
|
+
return nil unless record_data
|
|
159
|
+
|
|
135
160
|
DBF::Record.new(record_data, record_context)
|
|
136
161
|
end
|
|
137
162
|
|
|
@@ -142,14 +167,11 @@ module DBF
|
|
|
142
167
|
#
|
|
143
168
|
# @param [optional String, IO] path_or_io String path, IO-like object, or nil for STDOUT
|
|
144
169
|
def to_csv(path_or_io = nil)
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
170
|
+
if path_or_io.is_a?(String)
|
|
171
|
+
File.open(path_or_io, 'w') { |file| write_csv(file) }
|
|
172
|
+
else
|
|
173
|
+
write_csv(path_or_io || $stdout)
|
|
149
174
|
end
|
|
150
|
-
csv = CSV.new(io, force_quotes: true)
|
|
151
|
-
csv << column_names
|
|
152
|
-
each { |record| csv << record.to_a }
|
|
153
175
|
end
|
|
154
176
|
|
|
155
177
|
# Human readable version description
|
|
@@ -176,6 +198,37 @@ module DBF
|
|
|
176
198
|
|
|
177
199
|
private
|
|
178
200
|
|
|
201
|
+
def write_csv(io) # :nodoc:
|
|
202
|
+
csv = CSV.new(io, force_quotes: true)
|
|
203
|
+
csv << column_names.map { |name| csv_safe_value(name) }
|
|
204
|
+
each { |record| csv << record.to_a.map { |value| csv_safe_value(value) } }
|
|
205
|
+
end
|
|
206
|
+
|
|
207
|
+
# Neutralizes spreadsheet formula injection (CWE-1236) on CSV export by
|
|
208
|
+
# prefixing a single quote to string cells that begin with a formula
|
|
209
|
+
# trigger character. Non-string values (numbers, dates, booleans) are
|
|
210
|
+
# returned unchanged. The leading byte is compared numerically so that a
|
|
211
|
+
# value whose bytes are invalid in its encoding cannot raise here.
|
|
212
|
+
def csv_safe_value(value) # :nodoc:
|
|
213
|
+
return value unless value.is_a?(::String)
|
|
214
|
+
|
|
215
|
+
value = csv_compatible(value)
|
|
216
|
+
return value unless CSV_FORMULA_TRIGGERS.include?(value.getbyte(0))
|
|
217
|
+
|
|
218
|
+
quote = +"'"
|
|
219
|
+
quote.force_encoding(value.encoding) + value
|
|
220
|
+
end
|
|
221
|
+
|
|
222
|
+
# A row is written as a single string, so a binary (General/OLE) or
|
|
223
|
+
# invalidly encoded cell would raise Encoding::CompatibilityError when
|
|
224
|
+
# combined with text cells. Represent those bytes instead of raising.
|
|
225
|
+
def csv_compatible(value) # :nodoc:
|
|
226
|
+
return value if value.ascii_only?
|
|
227
|
+
return value if value.valid_encoding? && value.encoding != Encoding::BINARY
|
|
228
|
+
|
|
229
|
+
value.dup.force_encoding(Encoding::UTF_8).scrub('?')
|
|
230
|
+
end
|
|
231
|
+
|
|
179
232
|
def version_config
|
|
180
233
|
@version_config ||= VersionConfig.new(version)
|
|
181
234
|
end
|
data/lib/dbf/version.rb
CHANGED
data/lib/dbf/version_config.rb
CHANGED
|
@@ -68,11 +68,22 @@ module DBF
|
|
|
68
68
|
end
|
|
69
69
|
end
|
|
70
70
|
|
|
71
|
+
# Returns the Column.new arguments for the next column descriptor, or nil
|
|
72
|
+
# when the file ends mid-descriptor. Unpacking a short descriptor would
|
|
73
|
+
# otherwise raise ArgumentError ("x outside of string").
|
|
71
74
|
def read_column_args(table, io)
|
|
75
|
+
size, format, defaults = column_layout
|
|
76
|
+
data = io.read(size)
|
|
77
|
+
return nil unless data && data.bytesize == size
|
|
78
|
+
|
|
79
|
+
[table, *data.unpack(format), *defaults]
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
def column_layout # :nodoc:
|
|
72
83
|
case version
|
|
73
|
-
when '02' then [
|
|
74
|
-
when '04', '8c' then [
|
|
75
|
-
else [
|
|
84
|
+
when '02' then [header_size * 2, 'A11 a C', [0]]
|
|
85
|
+
when '04', '8c' then [48, 'A32 a C C x13', []]
|
|
86
|
+
else [header_size, 'A11 a x4 C2', []]
|
|
76
87
|
end
|
|
77
88
|
end
|
|
78
89
|
end
|
data/lib/dbf.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: dbf
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 5.
|
|
4
|
+
version: 5.5.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Keith Morrison
|
|
@@ -13,16 +13,16 @@ dependencies:
|
|
|
13
13
|
name: csv
|
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|
|
15
15
|
requirements:
|
|
16
|
-
- - "
|
|
16
|
+
- - "~>"
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
|
-
version: '
|
|
18
|
+
version: '3.3'
|
|
19
19
|
type: :runtime
|
|
20
20
|
prerelease: false
|
|
21
21
|
version_requirements: !ruby/object:Gem::Requirement
|
|
22
22
|
requirements:
|
|
23
|
-
- - "
|
|
23
|
+
- - "~>"
|
|
24
24
|
- !ruby/object:Gem::Version
|
|
25
|
-
version: '
|
|
25
|
+
version: '3.3'
|
|
26
26
|
description: A small fast library for reading dBase, xBase, Clipper and FoxPro database
|
|
27
27
|
files.
|
|
28
28
|
email: keithm@infused.org
|
|
@@ -44,6 +44,7 @@ files:
|
|
|
44
44
|
- lib/dbf/database/foxpro.rb
|
|
45
45
|
- lib/dbf/encoder.rb
|
|
46
46
|
- lib/dbf/encodings.rb
|
|
47
|
+
- lib/dbf/errors.rb
|
|
47
48
|
- lib/dbf/file_handler.rb
|
|
48
49
|
- lib/dbf/find.rb
|
|
49
50
|
- lib/dbf/header.rb
|
|
@@ -63,8 +64,12 @@ licenses:
|
|
|
63
64
|
- MIT
|
|
64
65
|
metadata:
|
|
65
66
|
rubygems_mfa_required: 'true'
|
|
67
|
+
homepage_uri: https://github.com/infused/dbf
|
|
66
68
|
source_code_uri: https://github.com/infused/dbf
|
|
67
69
|
changelog_uri: https://github.com/infused/dbf/blob/main/CHANGELOG.md
|
|
70
|
+
bug_tracker_uri: https://github.com/infused/dbf/issues
|
|
71
|
+
documentation_uri: https://dbf.infused.org
|
|
72
|
+
funding_uri: https://github.com/sponsors/infused
|
|
68
73
|
rdoc_options: []
|
|
69
74
|
require_paths:
|
|
70
75
|
- lib
|
|
@@ -79,7 +84,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
79
84
|
- !ruby/object:Gem::Version
|
|
80
85
|
version: '0'
|
|
81
86
|
requirements: []
|
|
82
|
-
rubygems_version:
|
|
87
|
+
rubygems_version: 3.6.9
|
|
83
88
|
specification_version: 4
|
|
84
89
|
summary: Read xBase files
|
|
85
90
|
test_files: []
|