dbf 5.3.0 → 5.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: d2d1ca393ae41262fd052c74ffe0ec497a963a2c036797686061312a0c492888
4
- data.tar.gz: 7e92c62ace7713b883b6b2e925c34f7a8114ebd6345b36a81f97fb47109711e2
3
+ metadata.gz: ff36c9c3d6b7a558cd5bf0f703f64f0cb7b22cbf6b4c52b843c855328a712a0a
4
+ data.tar.gz: ac0f725043e7dc9102a32381f6041e54ebd852a67a264778ae58b2a59430b01e
5
5
  SHA512:
6
- metadata.gz: f0f3b1b02ab53f505ab3efc1dff89d2ad1169698ef7d9443b3eec3b758b921ae2df51bf443d4e877ac072bdef87c547607d1d39d913a5c361b4c31e21fee4b07
7
- data.tar.gz: 6d826dd35aa6e80b671aeb804dc9991776851bbb9ca0ff5fb9ed6cd7139cbd40b0bad68327fc8f17b03b9ba2f2b77479331b7ad998c53e7ee32e5caee59241e6
6
+ metadata.gz: 269035188f9741d80575efca78ae0bda153c4f27c79dbe3f3e6221fc714be22eb2fb9ec39899dcfc9d489f76d60bfaf1f892d6eeaaeea80fa406f5f401e909d7
7
+ data.tar.gz: 6203ede51060bad9d193b15bb98d814aa870d99dd29d6eb8ddd4cb2b5e04f078f522fb7f3051befcbebe11be86a13307da38f9981bbcb704c5307ebdb16dd6a3
data/CHANGELOG.md CHANGED
@@ -1,7 +1,72 @@
1
1
  # Changelog
2
2
 
3
- ## main branch
4
-
3
+ ## 5.5.0
4
+
5
+ ### Added
6
+
7
+ - `DBF::Table.new` accepts any IO-like object responding to `#read` and `#seek` (`File`, `Tempfile`, …) in addition to a path or `StringIO`, matching what the README already documented; IO inputs are switched to binary mode, and memo auto-discovery works from the IO's path when it has one
8
+ - `DBF::Error` base class: all library errors (`FileNotFoundError`, `NoColumnsDefined`, `Column::LengthError`, `Column::InvalidNameError`) now inherit from it, so callers can `rescue DBF::Error` to catch anything the library raises
9
+ - `Table#filename` now works for any IO input with a path, not only `File`
10
+ - `DBF::Table.open`: same arguments as `.new`, but the block form yields the table, closes it when the block returns, and returns the block's value — the same contract as `File.open`
11
+ - CI: lint job running RuboCop and Reek on every push and pull request
12
+ - CI: test on Windows and macOS (Ruby 3.4) in addition to Linux, since much real-world DBF data originates on Windows/FoxPro systems
13
+ - CI: observe-only (non-blocking) test jobs for JRuby, TruffleRuby, and Ruby head
14
+ - CI: enforce a 99% minimum test coverage floor via SimpleCov
15
+ - Docs site (dbf.infused.org): YARD docs are now built and deployed by CI instead of being committed to the repository; the Pages workflow's deprecated v2/v3 actions (whose artifact backend was shut down in early 2025) are updated to current versions, repairing the broken deploy
16
+ - Dependabot updates for GitHub Actions and gems (weekly), a security policy (SECURITY.md) with private vulnerability reporting, and contributor documentation (CONTRIBUTING.md)
17
+ - Automated releases: pushing a `vX.Y.Z` tag now builds and publishes the gem to RubyGems.org from CI via Trusted Publishing (OIDC) — no long-lived API key. `bundler/gem_tasks` is loaded in the Rakefile, providing `rake build` / `rake release`. Git tags resume at v5.4.0; versions 4.0.0–5.3.0 were released without tags
18
+
19
+ - Ractor safety: all library constants are verified Ractor-shareable and reading/exporting tables inside non-main Ractors is covered by specs
20
+ - CLI: `-j` exports records as a JSON array and `-J` as JSON Lines (one record per line, streamed in constant memory); deleted records are skipped, control bytes are escaped by JSON itself, and binary cells are scrubbed instead of raising
21
+
22
+ ### Fixed
23
+
24
+ - `Table#to_csv(path)` closes the file it opens, so the CSV is fully flushed to disk when the method returns (previously the data could stay buffered until garbage collection)
25
+ - Visual FoxPro: a `.dbc` container supplying fewer long names than the table has columns no longer crashes column building; missing long names fall back to the table's own column names. Column rebuilding is also no longer O(n²)
26
+ - `Table#schema` validates the format against the documented list up front; a `NoMethodError` raised inside a valid schema generator is no longer misreported as "not a valid schema"
27
+ - FoxPro memo files shorter than their 512-byte header no longer crash memo reads (they return nil like other truncations)
28
+ - RuboCop config: restore the default `vendor/**/*` exclusion (overriding `Exclude` replaces the defaults), fixing a CI lint crash where RuboCop descended into cached vendored gems and tried to load plugins from their configs
29
+ - CLI: close the table when finished, so the DBF file can be deleted or replaced immediately afterwards on Windows (an open handle blocks deletion there)
30
+
31
+ ### Removed
32
+
33
+ - CodeClimate config and README badges: the service is sunset, and coverage is now enforced directly in CI
34
+
35
+ ### Changed
36
+
37
+ - Streaming reads: `Table#each` now reads records in 4 MB chunks of whole records instead of loading the entire record section into memory, so enumerating multi-gigabyte files (for example shapefile `.dbf` sidecars) uses bounded memory. On a 190 MB file, peak process memory drops from ~222 MB to ~35 MB with identical results. No API change; all malformed-file bounds are preserved
38
+ - Internal: `DBF::RecordContext` is now an immutable `Data` class instead of a `Struct`
39
+ - Internal: remove a redundant `Memo::Foxpro#initialize` and the duplicate `Header::HEADER_SIZE` constant (now sourced from `VersionConfig`)
40
+ - README: replace the stack of version-support notes with a compatibility table, point API docs at dbf.infused.org, and refer to LICENSE instead of inlining the full MIT text; copyright years updated through 2026
41
+ - Gemspec: add `homepage_uri`, `bug_tracker_uri`, `documentation_uri`, and `funding_uri` metadata (shown on the rubygems.org sidebar), and constrain the `csv` runtime dependency to `~> 3.3`
42
+ - `DBF::Column::NameError` is renamed to `DBF::Column::InvalidNameError` (it shadowed Ruby's `::NameError` without being one); the old constant remains as a deprecated alias
43
+ - Rescues in date/datetime decoding and FoxPro memo reads are narrowed to the specific expected errors, so genuine programmer errors surface instead of silently returning nil
44
+ - Declare `reek` and `simplecov` as direct development dependencies instead of relying on rubycritic's transitive dependencies
45
+ - Restrict `debug` and `ruby-lsp` to MRI so `bundle install` succeeds on JRuby/TruffleRuby
46
+ - Test matrix no longer fails fast, and a Ruby head regression no longer fails the build
47
+ - Update pinned `ruby/setup-ruby` action to v1.321.0
48
+
49
+ ## 5.4.0
50
+
51
+ - CLI: replace terminal control bytes in file-derived output so a crafted DBF cannot emit escape sequences to an interactive terminal; CSV and schema output are only filtered when writing to a terminal, so redirected exports are unchanged
52
+ - Security (CWE-248): write binary or invalidly encoded cells in `Table#to_csv` as representable text instead of raising an encoding error
53
+ - Security (CWE-248): replace unrepresentable column type bytes so a corrupt descriptor cannot raise when the schema is serialized to JSON
54
+ - Security (CWE-248): `Table#record` returns nil when the file ends after the delete flag instead of crashing on a nil record body
55
+ - Security (CWE-248): stop column parsing when the file ends mid-descriptor instead of raising `ArgumentError` from a short unpack
56
+ - Security (CWE-248): guard FoxPro memo-pointer decoding against a nil value from a truncated record instead of crashing on nil.unpack1
57
+ - Security (CWE-248): guard dBase IV memo reads against a nil/short block header instead of crashing on nil.unpack1
58
+ - Security (CWE-248): guard dBase III memo reads against a start block past EOF instead of crashing on a nil block
59
+ - Security (CWE-248): stop column parsing on a truncated descriptor instead of constructing an invalid column that crashes on a nil length
60
+ - Security (CWE-248): decode truncated numeric cells (Currency, AutoIncrement) to blank instead of raising an uncaught `nil` crash
61
+ - Security (CWE-248): handle truncated headers and missing column terminators gracefully instead of raising an uncaught `nil` crash during column parsing
62
+ - Security (CWE-400): bound FoxPro memo reads by the memo file size so a crafted memo size cannot force a ~4 GiB allocation
63
+ - Security (CWE-400): bound dBase IV memo reads by the memo file size so a crafted length field cannot force a ~4 GiB allocation
64
+ - Security (CWE-835): bound record iteration by the bytes actually read so a crafted `record_count` (or zero `record_length`) cannot cause an unbounded loop
65
+ - Security (CWE-789): bound the record read buffer by the file's actual size so a crafted header cannot force a multi-gigabyte allocation from a tiny file
66
+ - Security (CWE-400): resolve FoxPro `.dbc` tables by scanning the directory instead of `Dir.glob`, preventing glob brace-expansion CPU exhaustion from a crafted object name
67
+ - Security (CWE-22): confine Visual FoxPro `.dbc` table resolution to the database directory, preventing path traversal via a crafted container object name
68
+ - Security (CWE-1236): neutralize spreadsheet formula injection in `Table#to_csv` by prefixing a quote to string cells/headers starting with `= + - @`
69
+ - Security (CWE-94): escape table and column names when generating ActiveRecord/Sequel schemas, preventing Ruby code injection from crafted DBF header names
5
70
  - Add support for 8 more code pages (issue #98): Mazovia cp620 and Kamenický cp895 via vendored translation tables (new DBF::Encoder), plus macRoman, cp1255, cp1256, macCyrillic, macCentEuro and macGreek
6
71
  - Blank Visual FoxPro "T" (DateTime) columns now return nil instead of a Julian day-0 date
7
72
  - Blank "F" (Float) columns now return nil instead of 0.0, matching "N" (Number) behavior
data/LICENSE CHANGED
@@ -1,4 +1,4 @@
1
- Copyright (c) 2006-2024 Keith Morrison <keithm@infused.org>
1
+ Copyright (c) 2006-2026 Keith Morrison <keithm@infused.org>
2
2
 
3
3
  Permission is hereby granted, free of charge, to any person obtaining
4
4
  a copy of this software and associated documentation files (the
data/README.md CHANGED
@@ -2,36 +2,37 @@
2
2
 
3
3
  [![Version](https://img.shields.io/gem/v/dbf.svg?style=flat)](https://rubygems.org/gems/dbf)
4
4
  [![Build Status](https://github.com/infused/dbf/actions/workflows/build.yml/badge.svg)](https://github.com/infused/dbf/actions/workflows/build.yml)
5
- [![Code Quality](https://img.shields.io/codeclimate/maintainability/infused/dbf.svg?style=flat)](https://codeclimate.com/github/infused/dbf)
6
- [![Code Coverage](https://img.shields.io/codeclimate/c/infused/dbf.svg?style=flat)](https://codeclimate.com/github/infused/dbf)
7
5
  [![Total Downloads](https://img.shields.io/gem/dt/dbf.svg)](https://rubygems.org/gems/dbf/)
8
6
  [![License](https://img.shields.io/github/license/infused/dbf.svg)](https://github.com/infused/dbf)
9
7
 
10
8
  DBF is a small, fast Ruby library for reading dBase, xBase, Clipper, and FoxPro database files.
11
9
 
12
10
  * Project page: <https://github.com/infused/dbf>
13
- * API Documentation: <https://rdoc.info/gems/dbf>
11
+ * API Documentation: <https://dbf.infused.org>
14
12
  * Report bugs: <https://github.com/infused/dbf/issues>
15
13
  * Questions: Email <mailto:keithm@infused.org> and put DBF somewhere in the
16
14
  subject line
17
15
  * Change log: <https://github.com/infused/dbf/blob/main/CHANGELOG.md>
18
16
 
19
- NOTE: Beginning with version 5.2 we have dropped support for Ruby 3.2 and earlier.
20
-
21
- NOTE: Beginning with version 4.3 we have dropped support for Ruby 3.0 and earlier.
22
-
23
- NOTE: Beginning with version 4 we have dropped support for Ruby 2.0, 2.1, 2.2, and 2.3. If you need support for these older Rubies,
24
- please use 3.0.x (<https://github.com/infused/dbf/tree/3_stable>)
25
-
26
- NOTE: Beginning with version 3 we have dropped support for Ruby 1.8 and 1.9. If you need support for older Rubies,
27
- please use 2.0.x (<https://github.com/infused/dbf/tree/2_stable>)
28
-
29
17
  ## Compatibility
30
18
 
31
19
  DBF is tested to work with the following versions of Ruby:
32
20
 
33
21
  * Ruby 3.3.x, 3.4.x, 4.0.x
34
22
 
23
+ Older Rubies are supported by older DBF release lines:
24
+
25
+ | DBF version | Ruby support |
26
+ | ----------- | ------------ |
27
+ | 5.2+ | 3.3+ |
28
+ | 4.3 – 5.1 | 3.1+ |
29
+ | 4.0 – 4.2 | 2.4+ |
30
+ | 3.x ([3_stable](https://github.com/infused/dbf/tree/3_stable)) | 2.0 – 2.3 |
31
+ | 2.x ([2_stable](https://github.com/infused/dbf/tree/2_stable)) | 1.8 – 1.9 |
32
+
33
+ DBF is Ractor-friendly: all library constants are deeply frozen, so tables
34
+ can be opened, enumerated, and exported inside non-main Ractors.
35
+
35
36
  ## Installation
36
37
 
37
38
  Install the gem manually:
@@ -58,16 +59,35 @@ widgets = DBF::Table.new("widgets.dbf")
58
59
  Open a DBF file using an IO object:
59
60
 
60
61
  ```ruby
61
- data = File.open('widgets.dbf')
62
+ data = File.open('widgets.dbf', 'rb')
62
63
  widgets = DBF::Table.new(data)
63
64
  ```
64
65
 
66
+ All errors raised by the library inherit from DBF::Error, so you can rescue
67
+ the library as a unit:
68
+
69
+ ```ruby
70
+ begin
71
+ widgets = DBF::Table.new("widgets.dbf")
72
+ rescue DBF::Error => e
73
+ puts "Unable to read DBF file: #{e.message}"
74
+ end
75
+ ```
76
+
65
77
  Open a DBF by passing in raw data (wrap the raw data with StringIO):
66
78
 
67
79
  ```ruby
68
80
  widgets = DBF::Table.new(StringIO.new('raw binary data'))
69
81
  ```
70
82
 
83
+ Use the block form to close the table automatically, like File.open:
84
+
85
+ ```ruby
86
+ DBF::Table.open('widgets.dbf') do |table|
87
+ table.each { |record| puts record.name }
88
+ end
89
+ ```
90
+
71
91
  Enumerate all records
72
92
 
73
93
  ```ruby
@@ -287,13 +307,15 @@ table.name = 'my_table_name'
287
307
  A small command-line utility called dbf is installed with the gem.
288
308
 
289
309
  $ dbf -h
290
- usage: dbf [-h|-s|-a] filename
310
+ usage: dbf [-h|-s|-a|-c|-r|-j|-J] filename
291
311
  -h = print this message
292
- -v = print the version number
312
+ -v = print the DBF gem version
293
313
  -s = print summary information
294
314
  -a = create an ActiveRecord::Schema
295
- -r = create a Sequel Migration
315
+ -r = create a Sequel migration
296
316
  -c = export as CSV
317
+ -j = export as a JSON array
318
+ -J = export as JSON Lines (one record per line)
297
319
 
298
320
  Create an executable ActiveRecord schema:
299
321
 
@@ -307,6 +329,15 @@ Dump all records to a CSV file:
307
329
 
308
330
  dbf -c books.dbf > books.csv
309
331
 
332
+ Dump all records as JSON or JSON Lines:
333
+
334
+ dbf -j books.dbf > books.json
335
+ dbf -J books.dbf > books.jsonl
336
+
337
+ JSON Lines output is streamed record by record, so it works well for very
338
+ large files and pipelines (for example `dbf -J books.dbf | jq` or importing
339
+ into DuckDB).
340
+
310
341
  ## Reading a Visual Foxpro database (v8, v9)
311
342
 
312
343
  A special Database::Foxpro class is available to read Visual Foxpro container
@@ -374,25 +405,6 @@ Data type descriptions
374
405
 
375
406
  ## License
376
407
 
377
- Copyright (c) 2006-2024 Keith Morrison <<keithm@infused.org>>
378
-
379
- Permission is hereby granted, free of charge, to any person
380
- obtaining a copy of this software and associated documentation
381
- files (the "Software"), to deal in the Software without
382
- restriction, including without limitation the rights to use,
383
- copy, modify, merge, publish, distribute, sublicense, and/or sell
384
- copies of the Software, and to permit persons to whom the
385
- Software is furnished to do so, subject to the following
386
- conditions:
387
-
388
- The above copyright notice and this permission notice shall be
389
- included in all copies or substantial portions of the Software.
390
-
391
- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
392
- EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
393
- OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
394
- NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
395
- HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
396
- WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
397
- FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
398
- OTHER DEALINGS IN THE SOFTWARE.
408
+ Copyright (c) 2006-2026 Keith Morrison <<keithm@infused.org>>
409
+
410
+ Released under the MIT License. See [LICENSE](LICENSE) for the full text.
data/dbf.gemspec CHANGED
@@ -15,7 +15,11 @@ Gem::Specification.new do |s|
15
15
  s.require_paths = ['lib']
16
16
  s.required_ruby_version = '>= 3.3.0'
17
17
  s.metadata['rubygems_mfa_required'] = 'true'
18
+ s.metadata['homepage_uri'] = 'https://github.com/infused/dbf'
18
19
  s.metadata['source_code_uri'] = 'https://github.com/infused/dbf'
19
20
  s.metadata['changelog_uri'] = 'https://github.com/infused/dbf/blob/main/CHANGELOG.md'
20
- s.add_dependency 'csv'
21
+ s.metadata['bug_tracker_uri'] = 'https://github.com/infused/dbf/issues'
22
+ s.metadata['documentation_uri'] = 'https://dbf.infused.org'
23
+ s.metadata['funding_uri'] = 'https://github.com/sponsors/infused'
24
+ s.add_dependency 'csv', '~> 3.3'
21
25
  end
data/lib/dbf/cli.rb CHANGED
@@ -5,15 +5,61 @@ require 'optparse'
5
5
  module DBF
6
6
  class CLI
7
7
  USAGE = <<~HELP
8
- usage: dbf [-h|-s|-a|-c|-r] filename
8
+ usage: dbf [-h|-s|-a|-c|-r|-j|-J] filename
9
9
  -h = print this message
10
10
  -v = print the DBF gem version
11
11
  -s = print summary information
12
12
  -a = create an ActiveRecord::Schema
13
13
  -r = create a Sequel migration
14
14
  -c = export as CSV
15
+ -j = export as a JSON array
16
+ -J = export as JSON Lines (one record per line)
15
17
  HELP
16
18
 
19
+ # Bytes a terminal interprets as control or escape sequences. A crafted
20
+ # DBF can carry these in column names and record values, so they are
21
+ # replaced before file-derived text reaches an interactive terminal.
22
+ CONTROL_BYTES = /[\x00-\x1F\x7F]/n
23
+ # The same, but keeping CR and LF so CSV row separators survive.
24
+ CONTROL_BYTES_KEEPING_NEWLINES = /[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]/n
25
+
26
+ # Replaces terminal control bytes. Substitution happens on a byte copy so
27
+ # a value whose bytes are invalid in its encoding cannot raise here.
28
+ #
29
+ # @param value [Object]
30
+ # @param pattern [Regexp]
31
+ # @return [String]
32
+ def self.sanitize(value, pattern = CONTROL_BYTES)
33
+ string = value.to_s
34
+ string.b.gsub(pattern, '?').force_encoding(string.encoding)
35
+ end
36
+
37
+ # Wraps an IO so text written to an interactive terminal is stripped of
38
+ # control bytes. Redirected or piped output is never wrapped, so exported
39
+ # data is passed through unaltered.
40
+ class TerminalFilter
41
+ def initialize(io)
42
+ @io = io
43
+ end
44
+
45
+ def <<(data)
46
+ @io << CLI.sanitize(data, CONTROL_BYTES_KEEPING_NEWLINES)
47
+ self
48
+ end
49
+
50
+ def write(*data)
51
+ @io.write(*data.map { |datum| CLI.sanitize(datum, CONTROL_BYTES_KEEPING_NEWLINES) })
52
+ end
53
+
54
+ def method_missing(name, ...) # :nodoc:
55
+ @io.respond_to?(name) ? @io.send(name, ...) : super
56
+ end
57
+
58
+ def respond_to_missing?(name, include_private = false) # :nodoc:
59
+ @io.respond_to?(name, include_private) || super
60
+ end
61
+ end
62
+
17
63
  def self.run(argv, stdout: $stdout, stderr: $stderr)
18
64
  new(argv, stdout: stdout, stderr: stderr).run
19
65
  end
@@ -24,8 +70,17 @@ module DBF
24
70
  @stderr = stderr
25
71
  end
26
72
 
73
+ ACTIONS = {
74
+ 'a' => :print_ar_schema,
75
+ 'r' => :print_sequel_schema,
76
+ 's' => :print_summary,
77
+ 'c' => :print_csv,
78
+ 'j' => :print_json,
79
+ 'J' => :print_jsonl
80
+ }.freeze
81
+
27
82
  def run
28
- params = OptionParser.new.getopts(@argv, 'h', 's', 'a', 'c', 'r', 'v')
83
+ params = OptionParser.new.getopts(@argv, 'hsacrvjJ')
29
84
 
30
85
  if params['v']
31
86
  print_version
@@ -35,13 +90,8 @@ module DBF
35
90
  filename = @argv.shift
36
91
  return missing_filename unless filename
37
92
 
38
- action = %w[a r s c].find { |flag| params[flag] }
39
- case action
40
- when 'a' then print_ar_schema(filename)
41
- when 'r' then print_sequel_schema(filename)
42
- when 's' then print_summary(filename)
43
- when 'c' then print_csv(filename)
44
- end
93
+ action = ACTIONS.find { |flag, _method| params[flag] }&.last
94
+ send(action, filename) if action
45
95
  end
46
96
  0
47
97
  rescue DBF::FileNotFoundError => e
@@ -64,16 +114,23 @@ module DBF
64
114
  1
65
115
  end
66
116
 
117
+ # Always close the table when done: an open handle prevents deleting or
118
+ # replacing the file on Windows.
119
+ def with_table(filename, &) = DBF::Table.open(filename, &)
120
+
67
121
  def print_ar_schema(filename)
68
- @stdout.puts DBF::Table.new(filename).schema(:activerecord)
122
+ with_table(filename) { |table| @stdout.puts terminal_safe(table.schema(:activerecord)) }
69
123
  end
70
124
 
71
125
  def print_sequel_schema(filename)
72
- @stdout.puts DBF::Table.new(filename).schema(:sequel)
126
+ with_table(filename) { |table| @stdout.puts terminal_safe(table.schema(:sequel)) }
73
127
  end
74
128
 
75
129
  def print_summary(filename)
76
- table = DBF::Table.new(filename)
130
+ with_table(filename) { |table| write_summary(filename, table) }
131
+ end
132
+
133
+ def write_summary(filename, table)
77
134
  @stdout.puts
78
135
  @stdout.puts "Database: #{filename}"
79
136
  @stdout.puts "Type: (#{table.version}) #{table.version_description}"
@@ -84,12 +141,74 @@ module DBF
84
141
  @stdout.puts 'Name Type Length Decimal'
85
142
  @stdout.puts '-' * 78
86
143
  table.columns.each do |f|
87
- @stdout.puts format('%-16s %-10s %-10s %-10s', f.name, f.type, f.length, f.decimal)
144
+ # Column names and types come from the file. Always replace control
145
+ # bytes here: they are never valid in a name and would otherwise both
146
+ # emit escape sequences and break the column alignment below.
147
+ @stdout.puts format('%-16s %-10s %-10s %-10s', self.class.sanitize(f.name), self.class.sanitize(f.type), f.length, f.decimal)
88
148
  end
89
149
  end
90
150
 
91
151
  def print_csv(filename)
92
- DBF::Table.new(filename).to_csv(@stdout)
152
+ with_table(filename) { |table| table.to_csv(interactive? ? TerminalFilter.new(@stdout) : @stdout) }
153
+ end
154
+
155
+ # Streams a JSON array without materializing all records in memory.
156
+ # JSON string escaping makes the output terminal-safe by construction:
157
+ # control bytes are emitted as \uXXXX escapes.
158
+ def print_json(filename)
159
+ with_table(filename) { |table| write_json(table) }
160
+ end
161
+
162
+ def write_json(table)
163
+ first = true
164
+ @stdout.write('[')
165
+ each_present_record(table) do |record|
166
+ @stdout.write(first ? "\n" : ",\n")
167
+ @stdout.write(json_record(record))
168
+ first = false
169
+ end
170
+ @stdout.write("\n]\n")
171
+ end
172
+
173
+ # One JSON object per line (JSON Lines). Combined with chunked record
174
+ # reading this exports arbitrarily large files in constant memory.
175
+ def print_jsonl(filename)
176
+ with_table(filename) { |table| write_jsonl(table) }
177
+ end
178
+
179
+ def write_jsonl(table)
180
+ each_present_record(table) { |record| @stdout.write("#{json_record(record)}\n") }
181
+ end
182
+
183
+ # Deleted records have no attributes, so JSON export skips them.
184
+ def each_present_record(table, &)
185
+ table.each { |record| yield record if record }
186
+ end
187
+
188
+ def json_record(record)
189
+ JSON.generate(record.attributes.to_h { |key, value| [json_safe(key), json_safe(value)] })
190
+ end
191
+
192
+ # JSON.generate raises on binary or invalidly encoded strings; represent
193
+ # their bytes instead of raising, mirroring the CSV export behavior.
194
+ def json_safe(value)
195
+ return value unless value.is_a?(::String)
196
+
197
+ value.encode(Encoding::UTF_8, invalid: :replace, undef: :replace, replace: '?')
198
+ rescue Encoding::ConverterNotFoundError
199
+ value.dup.force_encoding(Encoding::UTF_8).scrub('?')
200
+ end
201
+
202
+ # Exported data is only filtered when it is going to a terminal, so
203
+ # redirecting or piping still produces byte-for-byte the original values.
204
+ def terminal_safe(text)
205
+ return text unless interactive?
206
+
207
+ self.class.sanitize(text, CONTROL_BYTES_KEEPING_NEWLINES)
208
+ end
209
+
210
+ def interactive?
211
+ @stdout.respond_to?(:tty?) && @stdout.tty?
93
212
  end
94
213
  end
95
214
  end
data/lib/dbf/column.rb CHANGED
@@ -2,12 +2,17 @@
2
2
 
3
3
  module DBF
4
4
  class Column
5
- class LengthError < StandardError
5
+ class LengthError < DBF::Error
6
6
  end
7
7
 
8
- class NameError < StandardError
8
+ class InvalidNameError < DBF::Error
9
9
  end
10
10
 
11
+ # Deprecated alias, kept for backward compatibility. It shadowed
12
+ # ::NameError (without being one), so it was renamed.
13
+ NameError = InvalidNameError
14
+ deprecate_constant :NameError
15
+
11
16
  attr_reader :name, :type, :length, :decimal
12
17
 
13
18
  # rubocop:disable Style/MutableConstant
@@ -38,7 +43,7 @@ module DBF
38
43
  def initialize(table, name, type, length, decimal)
39
44
  @table = table
40
45
  @name = clean(name)
41
- @type = type
46
+ @type = clean_type(type)
42
47
  @length = length
43
48
  @decimal = decimal
44
49
 
@@ -85,6 +90,14 @@ module DBF
85
90
  @table.encode_string(value.strip.split("\x00", 2).first || +'')
86
91
  end
87
92
 
93
+ # The column type is a single ASCII character. A corrupt file can supply
94
+ # any byte, which would otherwise stay binary and raise when serialized
95
+ # (e.g. to JSON), so replace anything unrepresentable.
96
+ def clean_type(value) # :nodoc:
97
+ type = value.to_s.dup.force_encoding(Encoding::UTF_8)
98
+ type.valid_encoding? ? type : type.scrub('?')
99
+ end
100
+
88
101
  def type_cast_class # :nodoc:
89
102
  @type_cast_class ||= begin
90
103
  klass = @length == 0 ? ColumnType::Nil : TYPE_CAST_CLASS[type.to_sym]
@@ -97,7 +110,7 @@ module DBF
97
110
  end
98
111
 
99
112
  def validate_name # :nodoc:
100
- raise NameError, 'column name cannot be empty' if @name.empty?
113
+ raise InvalidNameError, 'column name cannot be empty' if @name.empty?
101
114
  end
102
115
  end
103
116
  end
@@ -12,7 +12,15 @@ module DBF
12
12
  safe_seek do
13
13
  @data.seek(@version_config.header_size)
14
14
  [].tap do |columns|
15
- columns << Column.new(*@version_config.read_column_args(@table, @data)) until end_of_record?
15
+ until end_of_record?
16
+ args = @version_config.read_column_args(@table, @data)
17
+ # A descriptor truncated by EOF is returned as nil, or unpacks to
18
+ # a nil length; stop rather than constructing an invalid column
19
+ # (which would crash on nil < 0).
20
+ break if args.nil? || args[3].nil?
21
+
22
+ columns << Column.new(*args)
23
+ end
16
24
  end
17
25
  end
18
26
  end
@@ -20,7 +28,12 @@ module DBF
20
28
  private
21
29
 
22
30
  def end_of_record?
23
- safe_seek { @data.read(1).ord == 13 }
31
+ safe_seek do
32
+ byte = @data.read(1)
33
+ # A truncated file that ends before the 0x0D column terminator marks
34
+ # the end of the column list rather than crashing on nil.ord.
35
+ byte.nil? || byte.ord == 13
36
+ end
24
37
  end
25
38
 
26
39
  def safe_seek
@@ -20,6 +20,10 @@ module DBF
20
20
  end
21
21
 
22
22
  def decode(raw, &)
23
+ # A record truncated before this column yields a nil slice; treat it
24
+ # as blank rather than crashing in the type cast.
25
+ return blank_value if raw.nil?
26
+
23
27
  if skip_blank? && raw.count(' ') == raw.length
24
28
  blank_value
25
29
  else
@@ -49,7 +53,8 @@ module DBF
49
53
  class Currency < Base
50
54
  # @param value [String]
51
55
  def type_cast(value)
52
- (value.unpack1('q<') / 10_000.0).to_f
56
+ int = value.unpack1('q<')
57
+ int && (int / 10_000.0).to_f
53
58
  end
54
59
  end
55
60
 
@@ -64,6 +69,8 @@ module DBF
64
69
  # @param value [String]
65
70
  def type_cast(value)
66
71
  bits = value.unpack1('B*')
72
+ return nil unless bits && bits.length >= 32
73
+
67
74
  sign_multiplier = bits[0] == '0' ? -1 : 1
68
75
  bits[1, 31].to_i(2) * sign_multiplier
69
76
  end
@@ -105,7 +112,7 @@ module DBF
105
112
  # @param value [String]
106
113
  def type_cast(value)
107
114
  value.match?(/\d{8}/) && ::Date.strptime(value, '%Y%m%d')
108
- rescue StandardError
115
+ rescue ArgumentError, RangeError, TypeError
109
116
  nil
110
117
  end
111
118
  end
@@ -118,7 +125,7 @@ module DBF
118
125
 
119
126
  secs = (msecs / 1000).to_i
120
127
  ::DateTime.jd(days, (secs / 3600).to_i, (secs / 60).to_i % 60, secs % 60).to_time
121
- rescue StandardError
128
+ rescue ArgumentError, RangeError, TypeError
122
129
  nil
123
130
  end
124
131
  end
@@ -47,11 +47,25 @@ module DBF
47
47
  # @param name [String]
48
48
  # @return [String]
49
49
  def table_path(name)
50
- glob = File.join(@dirname, "#{name}.dbf")
51
- path = Dir.glob(glob, File::FNM_CASEFOLD).first
50
+ name = name.to_s
51
+ raise DBF::FileNotFoundError, "related table not found: #{name}" if name.empty? || name.include?("\x00")
52
+
53
+ # Treat the container-supplied name as an untrusted basename so that
54
+ # path separators and ".." cannot escape the database directory.
55
+ # Match case-insensitively by scanning the directory rather than
56
+ # globbing, so glob metacharacters (`{}`, `*`, `?`, `[]`) in the name
57
+ # cannot trigger brace-expansion CPU exhaustion.
58
+ target = "#{File.basename(name)}.dbf"
59
+ entry = Dir.children(@dirname).find { |child| child.casecmp?(target) }
60
+ path = entry && File.join(@dirname, entry)
52
61
 
53
62
  raise DBF::FileNotFoundError, "related table not found: #{name}" unless path && File.exist?(path)
54
63
 
64
+ # Defense in depth: confirm the resolved file really is inside the
65
+ # database directory before opening it.
66
+ contained = File.realpath(path).start_with?("#{File.realpath(@dirname)}#{File::SEPARATOR}")
67
+ raise DBF::FileNotFoundError, "related table not found: #{name}" unless contained
68
+
55
69
  path
56
70
  end
57
71
 
@@ -100,12 +114,10 @@ module DBF
100
114
  end
101
115
 
102
116
  def build_columns # :nodoc:
103
- columns = super
104
-
105
117
  # modify the column definitions to use the long names as the
106
118
  # columnname property is readonly, recreate the column definitions
107
- columns.map do |column|
108
- long_name = long_names[columns.index(column)]
119
+ super.each_with_index.map do |column, index|
120
+ long_name = long_names[index] || column.name
109
121
  Column.new(self, long_name, column.type, column.length, column.decimal)
110
122
  end
111
123
  end
data/lib/dbf/errors.rb ADDED
@@ -0,0 +1,14 @@
1
+ # frozen_string_literal: true
2
+
3
+ module DBF
4
+ # Base class for all errors raised by the DBF library, so callers can
5
+ # rescue DBF::Error to catch anything the library raises.
6
+ class Error < StandardError
7
+ end
8
+
9
+ class FileNotFoundError < Error
10
+ end
11
+
12
+ class NoColumnsDefined < Error
13
+ end
14
+ end
@@ -11,7 +11,12 @@ module DBF
11
11
  when String
12
12
  File.open(data, 'rb')
13
13
  else
14
- raise ArgumentError, 'data must be a file path or StringIO object'
14
+ raise ArgumentError, 'data must be a file path or an IO-like object responding to #read and #seek' unless data.respond_to?(:read) && data.respond_to?(:seek)
15
+
16
+ # DBF is a binary format; a File opened in text mode would corrupt
17
+ # reads on Windows.
18
+ data.binmode if data.respond_to?(:binmode)
19
+ data
15
20
  end
16
21
  rescue Errno::ENOENT
17
22
  raise DBF::FileNotFoundError, "file not found: #{data}"
@@ -19,17 +24,23 @@ module DBF
19
24
 
20
25
  def open_memo(data, memo, memo_class, version)
21
26
  if memo
22
- meth = memo.is_a?(StringIO) ? :new : :open
27
+ meth = memo.is_a?(String) ? :open : :new
23
28
  memo_class.send(meth, memo, version)
24
- elsif !data.is_a?(StringIO)
25
- path = Dir.glob(memo_search_path(data)).first
26
- path && memo_class.open(path, version)
29
+ elsif (path = data_path(data))
30
+ found = Dir.glob(memo_search_path(path)).first
31
+ found && memo_class.open(found, version)
27
32
  end
28
33
  end
29
34
 
30
- def memo_search_path(io)
31
- dirname = File.dirname(io)
32
- basename = File.basename(io, '.*')
35
+ def data_path(data)
36
+ return data if data.is_a?(String)
37
+
38
+ data.path if data.respond_to?(:path)
39
+ end
40
+
41
+ def memo_search_path(path)
42
+ dirname = File.dirname(path)
43
+ basename = File.basename(path, '.*')
33
44
  "#{dirname}/#{basename}*.{fpt,FPT,dbt,DBT}"
34
45
  end
35
46
  end
data/lib/dbf/header.rb CHANGED
@@ -5,6 +5,9 @@ module DBF
5
5
  attr_reader :version, :record_count, :header_length, :record_length, :encoding_key, :encoding
6
6
 
7
7
  def initialize(data)
8
+ # Pad a nil or truncated header read so unpacking a short file yields
9
+ # empty values instead of raising.
10
+ data = data.to_s.b.ljust(VersionConfig::DBASE3_HEADER_SIZE, "\x00")
8
11
  @version = data.unpack1('H2')
9
12
  @encoding_key = nil
10
13
  @encoding = nil
@@ -7,7 +7,12 @@ module DBF
7
7
  data.seek offset(start_block)
8
8
  memo_string = +''
9
9
  loop do
10
- block = data.read(BLOCK_SIZE).gsub(/(\000|\032)/, '')
10
+ block = data.read(BLOCK_SIZE)
11
+ # A start block past EOF yields nil; return what we have rather
12
+ # than crashing on nil.gsub.
13
+ break if block.nil?
14
+
15
+ block = block.gsub(/(\000|\032)/, '')
11
16
  memo_string << block
12
17
  break if block.size < BLOCK_SIZE
13
18
  end
@@ -5,7 +5,21 @@ module DBF
5
5
  class Dbase4 < Base
6
6
  def build_memo(start_block) # :nodoc:
7
7
  data.seek offset(start_block)
8
- data.read(data.read(BLOCK_HEADER_SIZE).unpack1('x4L'))
8
+
9
+ # A start block past EOF yields a nil/short header; return nil rather
10
+ # than crashing on nil.unpack1.
11
+ header = data.read(BLOCK_HEADER_SIZE)
12
+ return nil unless header && header.bytesize == BLOCK_HEADER_SIZE
13
+
14
+ length = header.unpack1('x4L')
15
+
16
+ # Bound the read by the bytes remaining so a crafted 32-bit length
17
+ # field cannot force a ~4 GiB allocation from a small memo file.
18
+ remaining = data.size - data.pos
19
+ length = remaining if length > remaining
20
+ return nil if length <= 0
21
+
22
+ data.read(length)
9
23
  end
10
24
  end
11
25
  end
@@ -5,35 +5,40 @@ module DBF
5
5
  class Foxpro < Base
6
6
  FPT_HEADER_SIZE = 512
7
7
 
8
- def initialize(data, version)
9
- @data = data
10
- super
11
- end
12
-
13
8
  def build_memo(start_block) # :nodoc:
14
- @data.seek offset(start_block)
15
- memo_type, memo_size, memo_string = @data.read(block_size).unpack('NNa*')
16
- return nil unless memo_type == 1 && memo_size > 0
9
+ data.seek offset(start_block)
10
+ block = data.read(block_size)
11
+ return nil unless block
12
+
13
+ # memo_size is nil when the block header is truncated (< 8 bytes)
14
+ memo_type, memo_size, memo_string = block.unpack('NNa*')
15
+ return nil unless memo_type == 1 && memo_size.to_i.positive?
17
16
 
18
17
  read_memo_content(memo_string, memo_size)
19
- rescue StandardError
18
+ rescue IOError, SystemCallError, RangeError
20
19
  nil
21
20
  end
22
21
 
23
22
  private
24
23
 
25
24
  def read_memo_content(memo_string, memo_size) # :nodoc:
26
- if memo_size > block_content_size
27
- memo_string << @data.read(content_size(memo_size))
28
- else
29
- memo_string[0, memo_size]
30
- end
25
+ return memo_string[0, memo_size] unless memo_size > block_content_size
26
+
27
+ # Bound the read by the bytes remaining so a crafted 32-bit memo_size
28
+ # cannot force a ~4 GiB allocation from a small memo file.
29
+ length = content_size(memo_size)
30
+ remaining = data.size - data.pos
31
+ length = remaining if length > remaining
32
+ memo_string << data.read(length) if length.positive?
33
+ memo_string
31
34
  end
32
35
 
33
36
  def block_size # :nodoc:
34
37
  @block_size ||= begin
35
- @data.rewind
36
- @data.read(FPT_HEADER_SIZE).unpack1('x6n') || 0
38
+ data.rewind
39
+ header = data.read(FPT_HEADER_SIZE)
40
+ # A header shorter than 8 bytes cannot contain the block size field
41
+ header && header.bytesize >= 8 ? header.unpack1('x6n') : 0
37
42
  end
38
43
  end
39
44
  end
data/lib/dbf/record.rb CHANGED
@@ -20,9 +20,7 @@ module DBF
20
20
  # @param [DBF::Record] other
21
21
  # @return [Boolean]
22
22
  def ==(other)
23
- attributes == other.attributes
24
- rescue NoMethodError
25
- false
23
+ other.respond_to?(:attributes) && attributes == other.attributes
26
24
  end
27
25
 
28
26
  # Reads attributes by column name
@@ -83,7 +81,9 @@ module DBF
83
81
 
84
82
  def decode_memo_value(raw) # :nodoc:
85
83
  memo = @context.memo
86
- return nil unless memo
84
+ # A record truncated before the memo column yields a nil pointer; skip
85
+ # decoding rather than crashing on nil.unpack1.
86
+ return nil unless memo && raw
87
87
 
88
88
  version = @context.version
89
89
  raw = raw.unpack1('V') if version == '30' || version == '31'
@@ -1,5 +1,6 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module DBF
4
- RecordContext = Struct.new(:columns, :version, :memo, :column_offsets, keyword_init: true)
4
+ # Immutable per-table state shared by every record during decoding
5
+ RecordContext = Data.define(:columns, :version, :memo, :column_offsets)
5
6
  end
@@ -2,34 +2,75 @@
2
2
 
3
3
  module DBF
4
4
  class RecordIterator
5
- def initialize(data, context, header_length, record_length, record_count)
5
+ # Records are read in chunks of whole records totalling roughly this many
6
+ # bytes, so enumerating a multi-gigabyte file (for example a shapefile
7
+ # sidecar) needs only chunk-sized memory instead of the entire record
8
+ # section at once.
9
+ CHUNK_SIZE = 4 * 1024 * 1024
10
+
11
+ def initialize(data, context, header_length, record_length, record_count, chunk_size: CHUNK_SIZE)
6
12
  @data = data
7
13
  @context = context
8
14
  @header_length = header_length
9
15
  @record_length = record_length
10
16
  @record_count = record_count
17
+ @chunk_size = chunk_size
18
+ end
19
+
20
+ def each(&)
21
+ return enum_for(:each) unless block_given?
22
+
23
+ # A record_length of 0 from a crafted header cannot drive an unbounded
24
+ # loop: capacity is 0 and enumeration ends immediately.
25
+ remaining = record_capacity
26
+ @data.seek(@header_length)
27
+
28
+ while remaining.positive?
29
+ wanted = [per_chunk, remaining].min
30
+ buffer = @data.read(wanted * @record_length)
31
+ break unless buffer
32
+
33
+ whole_records = buffer.bytesize / @record_length
34
+ break if whole_records.zero?
35
+
36
+ yield_chunk(buffer, whole_records, &)
37
+ remaining -= whole_records
38
+
39
+ # A short read means the file ended earlier than the header promised
40
+ break if whole_records < wanted
41
+ end
11
42
  end
12
43
 
13
- def each
14
- buf = read_buffer
15
- return unless buf
44
+ private
45
+
46
+ # Whole records per read; at least one so a record larger than the chunk
47
+ # size still makes progress. Only called when record_length is positive.
48
+ def per_chunk
49
+ @per_chunk ||= [@chunk_size / @record_length, 1].max
50
+ end
16
51
 
52
+ def yield_chunk(buffer, count)
17
53
  pos = 0
18
- @record_count.times do
19
- if buf.getbyte(pos) == 0x2A
54
+ count.times do
55
+ if buffer.getbyte(pos) == 0x2A
20
56
  yield nil
21
57
  else
22
- yield Record.new(buf, @context, pos + 1)
58
+ yield Record.new(buffer, @context, pos + 1)
23
59
  end
24
60
  pos += @record_length
25
61
  end
26
62
  end
27
63
 
28
- private
64
+ # Bound enumeration by the bytes actually available so a crafted header
65
+ # (huge record_length * record_count) cannot force reads past the real
66
+ # file size, while record_count still caps a file with trailing garbage.
67
+ def record_capacity
68
+ return 0 unless @record_length.positive?
29
69
 
30
- def read_buffer
31
- @data.seek(@header_length)
32
- @data.read(@record_length * @record_count)
70
+ available = @data.size - @header_length
71
+ return 0 if available.negative?
72
+
73
+ [@record_count, available / @record_length].min
33
74
  end
34
75
  end
35
76
  end
data/lib/dbf/schema.rb CHANGED
@@ -43,10 +43,9 @@ module DBF
43
43
  # @param table_only [Boolean]
44
44
  # @return [String]
45
45
  def schema(format = :activerecord, table_only: false)
46
- schema_method_name = schema_name(format)
47
- send(schema_method_name, table_only: table_only)
48
- rescue NameError
49
- raise ArgumentError, ":#{format} is not a valid schema. Valid schemas are: #{FORMATS.join(', ')}."
46
+ raise ArgumentError, ":#{format} is not a valid schema. Valid schemas are: #{FORMATS.join(', ')}." unless FORMATS.include?(format.to_s.to_sym)
47
+
48
+ send(schema_name(format), table_only: table_only)
50
49
  end
51
50
 
52
51
  def schema_name(format) # :nodoc:
@@ -55,7 +54,7 @@ module DBF
55
54
 
56
55
  def activerecord_schema(*) # :nodoc:
57
56
  output = +"ActiveRecord::Schema.define do\n"
58
- output << " create_table \"#{name}\" do |t|\n"
57
+ output << " create_table #{name.to_s.inspect} do |t|\n"
59
58
  columns.each do |column|
60
59
  output << " t.column #{activerecord_schema_definition(column)}"
61
60
  end
@@ -66,7 +65,7 @@ module DBF
66
65
  def sequel_schema(table_only: false) # :nodoc:
67
66
  output = +''
68
67
  output << "Sequel.migration do\n change do\n " unless table_only
69
- output << " create_table(:#{name}) do\n"
68
+ output << " create_table(#{name.to_s.to_sym.inspect}) do\n"
70
69
  columns.each do |column|
71
70
  output << " column #{sequel_schema_definition(column)}"
72
71
  end
@@ -84,7 +83,7 @@ module DBF
84
83
  # @param column [DBF::Column]
85
84
  # @return [String]
86
85
  def activerecord_schema_definition(column)
87
- "\"#{column.underscored_name}\", #{schema_data_type(column, :activerecord)}\n"
86
+ "#{column.underscored_name.inspect}, #{schema_data_type(column, :activerecord)}\n"
88
87
  end
89
88
 
90
89
  # Sequel schema definition
@@ -92,7 +91,7 @@ module DBF
92
91
  # @param column [DBF::Column]
93
92
  # @return [String]
94
93
  def sequel_schema_definition(column)
95
- ":#{column.underscored_name}, #{schema_data_type(column, :sequel)}\n"
94
+ "#{column.underscored_name.to_sym.inspect}, #{schema_data_type(column, :sequel)}\n"
96
95
  end
97
96
 
98
97
  def schema_data_type(column, format = :activerecord) # :nodoc:
data/lib/dbf/table.rb CHANGED
@@ -1,11 +1,9 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module DBF
4
- class FileNotFoundError < StandardError
5
- end
6
-
7
- class NoColumnsDefined < StandardError
8
- end
4
+ # Leading bytes that make a spreadsheet treat a CSV cell as a formula:
5
+ # "=", "+", "-", "@", tab, and carriage return.
6
+ CSV_FORMULA_TRIGGERS = [0x3D, 0x2B, 0x2D, 0x40, 0x09, 0x0D].freeze
9
7
 
10
8
  # DBF::Table is the primary interface to a single DBF file and provides
11
9
  # methods for enumerating and searching the records.
@@ -33,6 +31,9 @@ module DBF
33
31
  # # working with a dbf in memory
34
32
  # table = DBF::Table.new StringIO.new(dbf_data)
35
33
  #
34
+ # # working with an open IO object
35
+ # table = DBF::Table.new File.open('data.dbf', 'rb')
36
+ #
36
37
  # # working with a dbf and memo in memory
37
38
  # table = DBF::Table.new StringIO.new(dbf_data), StringIO.new(memo_data)
38
39
  #
@@ -40,8 +41,28 @@ module DBF
40
41
  # table = DBF::Table.new 'data.dbf', nil, 'cp437'
41
42
  # table = DBF::Table.new 'data.dbf', 'memo.dbt', Encoding::US_ASCII
42
43
  #
43
- # @param data [String, StringIO] data Path to the dbf file or a StringIO object
44
- # @param memo [optional String, StringIO] memo Path to the memo file or a StringIO object
44
+ # Opens a table like .new, but when given a block, yields the table,
45
+ # closes it when the block returns, and returns the block's value
46
+ # the same contract as File.open.
47
+ #
48
+ # DBF::Table.open('data.dbf') do |table|
49
+ # table.each { |record| ... }
50
+ # end
51
+ #
52
+ # Takes the same arguments as .new. Without a block, equivalent to .new.
53
+ def self.open(data, memo = nil, encoding = nil, name: nil)
54
+ table = new(data, memo, encoding, name: name)
55
+ return table unless block_given?
56
+
57
+ begin
58
+ yield table
59
+ ensure
60
+ table.close
61
+ end
62
+ end
63
+
64
+ # @param data [String, StringIO, IO] data Path to the dbf file or an IO-like object
65
+ # @param memo [optional String, StringIO, IO] memo Path to the memo file or an IO-like object
45
66
  # @param encoding [optional String, Encoding] encoding Name of the encoding or an Encoding object
46
67
  def initialize(data, memo = nil, encoding = nil, name: nil)
47
68
  @data = FileHandler.open_data(data)
@@ -106,7 +127,7 @@ module DBF
106
127
 
107
128
  # @return [String]
108
129
  def filename
109
- File.basename(@data.path) if @data.is_a?(File)
130
+ File.basename(@data.path) if @data.respond_to?(:path)
110
131
  end
111
132
 
112
133
  # @return [TrueClass, FalseClass]
@@ -132,6 +153,10 @@ module DBF
132
153
  return nil if deleted_record?
133
154
 
134
155
  record_data = @data.read(record_length)
156
+ # A file that ends immediately after the delete flag has no record body;
157
+ # treat it as absent rather than building a Record over nil data.
158
+ return nil unless record_data
159
+
135
160
  DBF::Record.new(record_data, record_context)
136
161
  end
137
162
 
@@ -142,14 +167,11 @@ module DBF
142
167
  #
143
168
  # @param [optional String, IO] path_or_io String path, IO-like object, or nil for STDOUT
144
169
  def to_csv(path_or_io = nil)
145
- io = case path_or_io
146
- when nil then $stdout
147
- when String then File.open(path_or_io, 'w')
148
- else path_or_io
170
+ if path_or_io.is_a?(String)
171
+ File.open(path_or_io, 'w') { |file| write_csv(file) }
172
+ else
173
+ write_csv(path_or_io || $stdout)
149
174
  end
150
- csv = CSV.new(io, force_quotes: true)
151
- csv << column_names
152
- each { |record| csv << record.to_a }
153
175
  end
154
176
 
155
177
  # Human readable version description
@@ -176,6 +198,37 @@ module DBF
176
198
 
177
199
  private
178
200
 
201
+ def write_csv(io) # :nodoc:
202
+ csv = CSV.new(io, force_quotes: true)
203
+ csv << column_names.map { |name| csv_safe_value(name) }
204
+ each { |record| csv << record.to_a.map { |value| csv_safe_value(value) } }
205
+ end
206
+
207
+ # Neutralizes spreadsheet formula injection (CWE-1236) on CSV export by
208
+ # prefixing a single quote to string cells that begin with a formula
209
+ # trigger character. Non-string values (numbers, dates, booleans) are
210
+ # returned unchanged. The leading byte is compared numerically so that a
211
+ # value whose bytes are invalid in its encoding cannot raise here.
212
+ def csv_safe_value(value) # :nodoc:
213
+ return value unless value.is_a?(::String)
214
+
215
+ value = csv_compatible(value)
216
+ return value unless CSV_FORMULA_TRIGGERS.include?(value.getbyte(0))
217
+
218
+ quote = +"'"
219
+ quote.force_encoding(value.encoding) + value
220
+ end
221
+
222
+ # A row is written as a single string, so a binary (General/OLE) or
223
+ # invalidly encoded cell would raise Encoding::CompatibilityError when
224
+ # combined with text cells. Represent those bytes instead of raising.
225
+ def csv_compatible(value) # :nodoc:
226
+ return value if value.ascii_only?
227
+ return value if value.valid_encoding? && value.encoding != Encoding::BINARY
228
+
229
+ value.dup.force_encoding(Encoding::UTF_8).scrub('?')
230
+ end
231
+
179
232
  def version_config
180
233
  @version_config ||= VersionConfig.new(version)
181
234
  end
data/lib/dbf/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module DBF
4
- VERSION = '5.3.0'
4
+ VERSION = '5.5.0'
5
5
  end
@@ -68,11 +68,22 @@ module DBF
68
68
  end
69
69
  end
70
70
 
71
+ # Returns the Column.new arguments for the next column descriptor, or nil
72
+ # when the file ends mid-descriptor. Unpacking a short descriptor would
73
+ # otherwise raise ArgumentError ("x outside of string").
71
74
  def read_column_args(table, io)
75
+ size, format, defaults = column_layout
76
+ data = io.read(size)
77
+ return nil unless data && data.bytesize == size
78
+
79
+ [table, *data.unpack(format), *defaults]
80
+ end
81
+
82
+ def column_layout # :nodoc:
72
83
  case version
73
- when '02' then [table, *io.read(header_size * 2).unpack('A11 a C'), 0]
74
- when '04', '8c' then [table, *io.read(48).unpack('A32 a C C x13')]
75
- else [table, *io.read(header_size).unpack('A11 a x4 C2')]
84
+ when '02' then [header_size * 2, 'A11 a C', [0]]
85
+ when '04', '8c' then [48, 'A32 a C C x13', []]
86
+ else [header_size, 'A11 a x4 C2', []]
76
87
  end
77
88
  end
78
89
  end
data/lib/dbf.rb CHANGED
@@ -7,6 +7,7 @@ require 'json'
7
7
  require 'time'
8
8
 
9
9
  require 'dbf/version'
10
+ require 'dbf/errors'
10
11
  require 'dbf/schema'
11
12
  require 'dbf/find'
12
13
  require 'dbf/record'
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: dbf
3
3
  version: !ruby/object:Gem::Version
4
- version: 5.3.0
4
+ version: 5.5.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Keith Morrison
@@ -13,16 +13,16 @@ dependencies:
13
13
  name: csv
14
14
  requirement: !ruby/object:Gem::Requirement
15
15
  requirements:
16
- - - ">="
16
+ - - "~>"
17
17
  - !ruby/object:Gem::Version
18
- version: '0'
18
+ version: '3.3'
19
19
  type: :runtime
20
20
  prerelease: false
21
21
  version_requirements: !ruby/object:Gem::Requirement
22
22
  requirements:
23
- - - ">="
23
+ - - "~>"
24
24
  - !ruby/object:Gem::Version
25
- version: '0'
25
+ version: '3.3'
26
26
  description: A small fast library for reading dBase, xBase, Clipper and FoxPro database
27
27
  files.
28
28
  email: keithm@infused.org
@@ -44,6 +44,7 @@ files:
44
44
  - lib/dbf/database/foxpro.rb
45
45
  - lib/dbf/encoder.rb
46
46
  - lib/dbf/encodings.rb
47
+ - lib/dbf/errors.rb
47
48
  - lib/dbf/file_handler.rb
48
49
  - lib/dbf/find.rb
49
50
  - lib/dbf/header.rb
@@ -63,8 +64,12 @@ licenses:
63
64
  - MIT
64
65
  metadata:
65
66
  rubygems_mfa_required: 'true'
67
+ homepage_uri: https://github.com/infused/dbf
66
68
  source_code_uri: https://github.com/infused/dbf
67
69
  changelog_uri: https://github.com/infused/dbf/blob/main/CHANGELOG.md
70
+ bug_tracker_uri: https://github.com/infused/dbf/issues
71
+ documentation_uri: https://dbf.infused.org
72
+ funding_uri: https://github.com/sponsors/infused
68
73
  rdoc_options: []
69
74
  require_paths:
70
75
  - lib
@@ -79,7 +84,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
79
84
  - !ruby/object:Gem::Version
80
85
  version: '0'
81
86
  requirements: []
82
- rubygems_version: 4.0.10
87
+ rubygems_version: 3.6.9
83
88
  specification_version: 4
84
89
  summary: Read xBase files
85
90
  test_files: []