duckling 0.2.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.claude/settings.json +46 -0
- data/AGENTS.md +208 -0
- data/Brewfile +3 -0
- data/CHANGELOG.md +110 -0
- data/CLAUDE.md +1 -0
- data/{ext/duckling/Cargo.lock → Cargo.lock} +20 -0
- data/Cargo.toml +9 -0
- data/README.md +216 -4
- data/Rakefile +231 -3
- data/docs/2026-07-01-roadmap.md +106 -0
- data/docs/tz-database-axis.md +454 -0
- data/ext/duckling/Cargo.toml +11 -2
- data/ext/duckling/src/lib.rs +411 -79
- data/ext/duckling/src/ruby_value.rs +104 -0
- data/lib/duckling/tzinfo_capabilities.rb +59 -0
- data/lib/duckling/version.rb +1 -1
- data/lib/duckling.rb +281 -1
- metadata +58 -7
data/README.md
CHANGED
|
@@ -18,13 +18,225 @@ gem install duckling
|
|
|
18
18
|
|
|
19
19
|
## Usage
|
|
20
20
|
|
|
21
|
-
|
|
21
|
+
```ruby
|
|
22
|
+
require "duckling"
|
|
23
|
+
|
|
24
|
+
Duckling.parse("tomorrow", locale: "en")
|
|
25
|
+
# =>
|
|
26
|
+
# [{ body: "tomorrow", start: 0, end: 8, dim: :time,
|
|
27
|
+
# value: { Time: { Single: { value: { Naive: { value: 2026-07-02 00:00:00 +0000, grain: :day } },
|
|
28
|
+
# values: [...] } } } }]
|
|
29
|
+
# (the date resolves relative to now; pass reference_time: for a fixed anchor)
|
|
30
|
+
```
|
|
22
31
|
|
|
23
|
-
|
|
32
|
+
`Duckling.parse` takes required text plus keyword options, and returns an
|
|
33
|
+
`Array` of entity `Hash`es (empty if nothing matched):
|
|
34
|
+
|
|
35
|
+
```ruby
|
|
36
|
+
Duckling.parse(text, locale: "en", dims: ["time"], reference_time: nil, with_latent: false, reference_zone: nil)
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
### Keyword arguments
|
|
40
|
+
|
|
41
|
+
- `locale:` (String, default `"en"`) — a `lang[-region]` tag, e.g. `"en"` or
|
|
42
|
+
`"en-GB"`. An unrecognized language or region raises `ArgumentError`.
|
|
43
|
+
- `dims:` (Array of String, default `["time"]`) — which dimensions to
|
|
44
|
+
extract. See "Supported dimensions" below. An unrecognized dimension name
|
|
45
|
+
raises `ArgumentError`.
|
|
46
|
+
- `reference_time:` (`Time`, default `nil`) — anchors relative expressions
|
|
47
|
+
like "tomorrow" or "next week". Its `utc_offset` is preserved into every
|
|
48
|
+
time result's `:value` — both wall-clock expressions ("tomorrow", "next
|
|
49
|
+
week") and offset-aware ones ("in one hour"), not flattened to UTC.
|
|
50
|
+
Defaults to the current UTC time; pass an explicit `Time` for deterministic
|
|
51
|
+
output. Anything else responding to `to_time` —
|
|
52
|
+
`ActiveSupport::TimeWithZone` (`Time.current`/`Time.zone.now`), stdlib
|
|
53
|
+
`DateTime`, etc. — is coerced to a `Time` automatically. A value that's
|
|
54
|
+
neither a `Time` nor `to_time`-able (e.g. a raw Integer Unix timestamp)
|
|
55
|
+
raises `TypeError`; wrap it in `Time.at(seconds)` first.
|
|
56
|
+
- `with_latent:` (Boolean, default `false`) — include ambiguous/latent
|
|
57
|
+
matches (e.g. a bare "morning") in the results.
|
|
58
|
+
- `reference_zone:` (String, default `nil`) — an IANA zone name, e.g.
|
|
59
|
+
`"America/New_York"`. Resolves each wall-clock result's UTC offset against
|
|
60
|
+
that zone on the result's *own* date, so a result before a DST transition
|
|
61
|
+
and one after it get different offsets instead of sharing
|
|
62
|
+
`reference_time:`'s single fixed one. An unknown identifier raises
|
|
63
|
+
`ArgumentError`; so does a `reference_time:` whose `utc_offset` disagrees
|
|
64
|
+
with the zone at that instant. It does not anchor the parse — see "Time
|
|
65
|
+
zone data" below.
|
|
66
|
+
|
|
67
|
+
There is no `Duckling::Error` class — invalid `locale:`/`dims:` values raise
|
|
68
|
+
plain `ArgumentError`, as do an unknown `reference_zone:` and a
|
|
69
|
+
`reference_time:` whose offset disagrees with it. The one named error a caller
|
|
70
|
+
is likely to meet is `Duckling::TZDataUnavailable`, for a host with no tz
|
|
71
|
+
database at all — see "Time zone data" below.
|
|
72
|
+
|
|
73
|
+
### Time zone data
|
|
74
|
+
|
|
75
|
+
`reference_zone:` resolves against [tzinfo][], which uses the `tzinfo-data`
|
|
76
|
+
gem when it is installed and the host's own zoneinfo files otherwise. This gem
|
|
77
|
+
does not depend on `tzinfo-data`, so by default you get the host's database.
|
|
78
|
+
|
|
79
|
+
That is usually what you want, and it is the faster of the two to start up.
|
|
80
|
+
Two cases where it is not:
|
|
81
|
+
|
|
82
|
+
- **Backward-compatibility names.** Debian and Ubuntu ship names like
|
|
83
|
+
`"US/Eastern"` in a separate `tzdata-legacy` package that is not installed
|
|
84
|
+
by default, so roughly a hundred valid IANA identifiers raise
|
|
85
|
+
`ArgumentError` on a stock host. Either `gem "tzinfo-data"` or
|
|
86
|
+
`apt install tzdata-legacy` restores them. The error message names whichever
|
|
87
|
+
database answered and how many identifiers it has, so you can tell this
|
|
88
|
+
apart from a typo.
|
|
89
|
+
- **No zoneinfo files at all**, as in a scratch or distroless container.
|
|
90
|
+
`reference_zone:` raises `Duckling::TZDataUnavailable` there, naming both
|
|
91
|
+
fixes. Add `gem "tzinfo-data"` to bundle the data with your app, where you
|
|
92
|
+
can also patch its vintage by bumping one gem, or install the system
|
|
93
|
+
`tzdata` package. Every other keyword works without a tz database.
|
|
94
|
+
|
|
95
|
+
`reference_zone:` reinterprets result offsets; it does not anchor the parse.
|
|
96
|
+
Given without `reference_time:`, a relative expression like `"tomorrow"` still
|
|
97
|
+
anchors on the machine-local clock rather than on "now" in that zone — pass a
|
|
98
|
+
`reference_time:` in the zone to anchor as well.
|
|
99
|
+
|
|
100
|
+
[tzinfo]: https://github.com/tzinfo/tzinfo
|
|
101
|
+
|
|
102
|
+
### Return value
|
|
103
|
+
|
|
104
|
+
Each entity in the returned array is a `Hash` with:
|
|
105
|
+
|
|
106
|
+
- `:body` (String) — the matched substring.
|
|
107
|
+
- `:start` / `:end` (Integer) — character offsets into the input text.
|
|
108
|
+
- `:dim` (Symbol) — the dimension, e.g. `:time`.
|
|
109
|
+
- `:latent` (Boolean) — present only when the match is latent.
|
|
110
|
+
- `:value` — every entity carries one; its shape depends on `:dim`.
|
|
111
|
+
|
|
112
|
+
Every dimension's `:value` is serde's externally-tagged representation of
|
|
113
|
+
the underlying Rust value, with all Hash keys symbolized — a one-pair
|
|
114
|
+
`Hash` keyed by a PascalCase tag matching the dimension (`{Numeral: 42.0}`,
|
|
115
|
+
`{Email: "user@example.com"}`, `{Url: {value: "...", domain: "..."}}`,
|
|
116
|
+
`{Temperature: {Value: {value: 37.0, unit: "celsius"}}}`). See
|
|
117
|
+
`test/duckling_parse_dimensions_test.rb` for the exact shape of every
|
|
118
|
+
dimension.
|
|
119
|
+
|
|
120
|
+
`:time` is the one dimension with additional nested tagging, since its
|
|
121
|
+
underlying leaf datetimes aren't serde-serializable as Ruby `Time`
|
|
122
|
+
objects and need patching after the fact (see `ext/duckling/src/lib.rs`'s
|
|
123
|
+
`patch_time_value`/`patch_time_point`):
|
|
124
|
+
|
|
125
|
+
```ruby
|
|
126
|
+
# a single point in time, e.g. "tomorrow"
|
|
127
|
+
{ Time: { Single: { value: { Naive: { value: 2026-07-02 00:00:00 +0000, grain: :day } },
|
|
128
|
+
values: [...] } } }
|
|
129
|
+
|
|
130
|
+
# an interval, e.g. "from 3pm to 5pm"
|
|
131
|
+
{ Time: { Interval: {
|
|
132
|
+
from: { Naive: { value: 2013-02-12 15:00:00 -0200, grain: :hour } },
|
|
133
|
+
to: { Naive: { value: 2013-02-12 18:00:00 -0200, grain: :hour } },
|
|
134
|
+
values: [...] } } }
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Every `TimePoint` (the primary `value:`, each `values:` recurrence entry,
|
|
138
|
+
and an interval's `from:`/`to:`) is tagged `Naive` (wall-clock, no
|
|
139
|
+
timezone assumption — e.g. "tomorrow", "5pm") or `Instant` (offset-aware —
|
|
140
|
+
e.g. "in one hour", "now"). `grain` is one of `second`, `minute`, `hour`,
|
|
141
|
+
`day`, `week`, `month`, `quarter`, `year`. The nested `value:` is always a
|
|
142
|
+
real Ruby `Time`, not a formatted string — its `utc_offset` matches
|
|
143
|
+
`reference_time:`'s (or UTC, if `reference_time:` was omitted). A holiday
|
|
144
|
+
match (e.g. "christmas") additionally carries `holidayBeta:` (a String)
|
|
145
|
+
alongside `Single`'s `value:`/`values:`.
|
|
146
|
+
|
|
147
|
+
**Gotcha:** an interval's `:to` is the *exclusive* boundary, not the
|
|
148
|
+
literal named time — `"from 3pm to 5pm"` resolves `:to` to `18:00`, not
|
|
149
|
+
`17:00`. This matches upstream [duckling](https://github.com/wafer-inc/duckling)
|
|
150
|
+
behavior.
|
|
151
|
+
|
|
152
|
+
**Gotcha:** an unbounded interval (e.g. `"after 3pm"`, `"until 5pm"`)
|
|
153
|
+
still carries both `:from` and `:to` keys — the missing bound is an
|
|
154
|
+
explicit `nil` value, not an absent key. Check `interval[:to].nil?`
|
|
155
|
+
(not `interval.key?(:to)`) to detect an unbounded endpoint.
|
|
156
|
+
|
|
157
|
+
### Supported dimensions
|
|
158
|
+
|
|
159
|
+
`dims:` accepts `time`, `number`, `ordinal`, `temperature`, `distance`,
|
|
160
|
+
`volume`, `quantity`, `amount-of-money`, `email`, `phone-number`, `url`,
|
|
161
|
+
`credit-card-number`, `time-grain`, and `duration` — every entity comes back
|
|
162
|
+
with a populated `:value` (see "Return value" above for the shape). An
|
|
163
|
+
unrecognized dimension name raises `ArgumentError`.
|
|
164
|
+
|
|
165
|
+
### Known limitation: bare comma-separated lists
|
|
166
|
+
|
|
167
|
+
A run of date/time expressions joined by bare commas, with nothing else
|
|
168
|
+
between them, collapses into a single entity — every date after the first
|
|
169
|
+
in that run is silently dropped:
|
|
170
|
+
|
|
171
|
+
```ruby
|
|
172
|
+
Duckling.parse("birthdays are march 3, march 9, april 12 and may 5", locale: "en")
|
|
173
|
+
.select { |r| r[:dim] == :time }
|
|
174
|
+
.map { |r| r[:value][:value] }
|
|
175
|
+
# => [2013-03-03 00:00:00 +0000, 2013-05-05 00:00:00 +0000]
|
|
176
|
+
# (march 9 and april 12 are silently dropped)
|
|
177
|
+
```
|
|
24
178
|
|
|
25
|
-
|
|
179
|
+
This is an upstream grammar/ranking behavior in the wrapped
|
|
180
|
+
[duckling](https://github.com/wafer-inc/duckling) engine, not something this
|
|
181
|
+
gem can work around. Joining
|
|
182
|
+
dates with "and", periods, or a name/label immediately before each date
|
|
183
|
+
avoids the collapse:
|
|
184
|
+
|
|
185
|
+
```ruby
|
|
186
|
+
Duckling.parse("march 3 and march 9 and april 12 and may 5", locale: "en")
|
|
187
|
+
.select { |r| r[:dim] == :time }
|
|
188
|
+
.map { |r| r[:value][:value] }
|
|
189
|
+
# => [2013-03-03 00:00:00 +0000, 2013-03-09 00:00:00 +0000, 2013-04-12 00:00:00 +0000, 2013-05-05 00:00:00 +0000]
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
See `test/duckling_comma_list_test.rb` for the full characterization,
|
|
193
|
+
including cases where the surviving value isn't even reliably the leftmost
|
|
194
|
+
date in the collapsed run.
|
|
195
|
+
|
|
196
|
+
## Performance
|
|
197
|
+
|
|
198
|
+
Benchmarked with [benchmark-ips](https://github.com/evanphx/benchmark-ips)
|
|
199
|
+
against `Duckling.parse`, including Magnus/Ruby conversion overhead (not
|
|
200
|
+
just the underlying Rust engine), plus GC pressure and threaded-worker-pool
|
|
201
|
+
throughput. See [`docs/benchmarks/`](docs/benchmarks/) for the latest
|
|
202
|
+
numbers, broken out by environment (GitHub Actions CI, Claude Code Web,
|
|
203
|
+
local dev) — results vary enough by machine that comparing across
|
|
204
|
+
environments is more meaningful than a single blended trend.
|
|
205
|
+
|
|
206
|
+
## Development
|
|
26
207
|
|
|
27
|
-
|
|
208
|
+
After checking out the repo, run `bin/setup` to install dependencies. Building
|
|
209
|
+
the native Rust extension requires a Rust toolchain; on macOS with Homebrew
|
|
210
|
+
installed, `bin/setup` installs it automatically via `brew bundle` and the
|
|
211
|
+
project's `Brewfile` (no-op if Homebrew isn't present). Then run
|
|
212
|
+
`rake compile` to build the extension before running `rake test`, or just run
|
|
213
|
+
`rake` (or `bundle exec rake`) with no arguments to lint, compile, and test in
|
|
214
|
+
order. You can also run `bin/console` for an interactive prompt that will
|
|
215
|
+
allow you to experiment.
|
|
216
|
+
|
|
217
|
+
`bin/setup` also seeds a `.env.local` file (from `.env.local.example`) with
|
|
218
|
+
`RB_SYS_CARGO_PROFILE=dev`, so local `rake compile` runs build the extension
|
|
219
|
+
in Cargo's dev profile by default — slower at runtime, but much faster to
|
|
220
|
+
compile while iterating. `.env.local` is gitignored and untouched by CI, so
|
|
221
|
+
CI and `rake release` still build the optimized release profile. Delete or
|
|
222
|
+
edit `.env.local` to opt back into a release-profile local build, or run
|
|
223
|
+
`bundle exec rake dev compile test` for a one-off dev-profile build without
|
|
224
|
+
`.env.local` in place.
|
|
225
|
+
|
|
226
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version: bump `Duckling::VERSION` in `version.rb`, merge that change to `main`, then run `bundle exec rake release` (or push a matching `vX.Y.Z` tag directly) to create and push the git tag. Pushing the tag triggers a GitHub Actions pipeline that re-runs CI as a gate, cross-compiles `x86_64-linux`/`x86_64-darwin` binary gems, verifies the tag matches `Duckling::VERSION`, builds and publishes the gems (source + both binary platforms) to [rubygems.org](https://rubygems.org), cuts a GitHub release, and opens a PR appending an entry to `CHANGELOG.md`.
|
|
227
|
+
|
|
228
|
+
`bin/benchmark` (or `bundle exec rake benchmark`) runs the `benchmark-ips`
|
|
229
|
+
suite locally and prints results to the console — no files written.
|
|
230
|
+
`bin/benchmark record` (or `rake benchmark:record`) additionally writes
|
|
231
|
+
`docs/benchmarks/<environment>/<version>.json` and regenerates
|
|
232
|
+
`docs/benchmarks/README.md`. `bin/benchmark record-pr` (or `rake
|
|
233
|
+
benchmark:record_pr`) does the same against a fresh branch off `origin/main`
|
|
234
|
+
and opens (and auto-merges) a PR via `gh` — this is what the release
|
|
235
|
+
pipeline runs automatically, and what you'd also run from a Claude Code Web
|
|
236
|
+
session or a local dev machine to contribute that environment's numbers
|
|
237
|
+
ahead of a release. `gh` needs to be installed (`bin/setup` does this via
|
|
238
|
+
the `Brewfile` on macOS) and authenticated (`gh auth login`) for the
|
|
239
|
+
`record-pr` variant.
|
|
28
240
|
|
|
29
241
|
## Contributing
|
|
30
242
|
|
data/Rakefile
CHANGED
|
@@ -1,15 +1,243 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require "bundler/gem_tasks"
|
|
4
|
+
require "dotenv"
|
|
4
5
|
require "minitest/test_task"
|
|
5
|
-
require "
|
|
6
|
+
require "rb_sys/extensiontask"
|
|
6
7
|
|
|
7
|
-
|
|
8
|
+
require_relative "cross_targets"
|
|
9
|
+
|
|
10
|
+
# Loads RB_SYS_CARGO_PROFILE=dev from .env.local when present (seeded by
|
|
11
|
+
# bin/setup from .env.local.example), so local compiles default to the dev
|
|
12
|
+
# profile without needing the :dev task below. .env.local is gitignored and
|
|
13
|
+
# never checked out in CI, so `bundle exec rake` there still builds release.
|
|
14
|
+
Dotenv.load(".env.local")
|
|
15
|
+
|
|
16
|
+
GEMSPEC = Gem::Specification.load("duckling.gemspec")
|
|
17
|
+
|
|
18
|
+
RbSys::ExtensionTask.new("duckling", GEMSPEC) do |ext|
|
|
8
19
|
ext.lib_dir = "lib/duckling"
|
|
20
|
+
|
|
21
|
+
# rake-compiler derives a native gem's required_ruby_version from the ABIs
|
|
22
|
+
# it cross-compiled against, and writes its own floor over the gemspec's.
|
|
23
|
+
# The gemspec's floor can be stricter, so state both bounds here: the
|
|
24
|
+
# gemspec's own requirement, plus the ceiling from cross_targets.rb.
|
|
25
|
+
ext.cross_compiling do |spec|
|
|
26
|
+
spec.required_ruby_version =
|
|
27
|
+
GEMSPEC.required_ruby_version.as_list + ["< #{CrossTargets::ABI_CEILING}"]
|
|
28
|
+
end
|
|
9
29
|
end
|
|
10
30
|
|
|
11
|
-
|
|
31
|
+
# rake-compiler always builds two things for this extension:
|
|
32
|
+
# - The real cross build for RUBY_TARGET.
|
|
33
|
+
# - An extra "local" build for the host Ruby. rake-compiler runs this
|
|
34
|
+
# local build even when cross_compile is off.
|
|
35
|
+
#
|
|
36
|
+
# Gem::PackageTask lists the local build's plain output path as a
|
|
37
|
+
# prerequisite of the final .gem file.
|
|
38
|
+
#
|
|
39
|
+
# Each rbsys/<platform> Docker image sets RUST_TARGET and
|
|
40
|
+
# CARGO_BUILD_TARGET in its environment. These variables make a plain
|
|
41
|
+
# `cargo build` target that image's platform. Every process in the
|
|
42
|
+
# container inherits them, including the local build.
|
|
43
|
+
#
|
|
44
|
+
# The Ruby that drives rake in each container is an x86_64 Linux Ruby.
|
|
45
|
+
# So the local build always makes a file named duckling.so.
|
|
46
|
+
#
|
|
47
|
+
# On x86_64-linux, this causes no problem. The local build's target is
|
|
48
|
+
# already correct.
|
|
49
|
+
#
|
|
50
|
+
# On x86_64-darwin and arm64-darwin, this causes no problem either.
|
|
51
|
+
# Those gems need duckling.bundle. They do not list duckling.so, so the
|
|
52
|
+
# local build is not a prerequisite and never runs.
|
|
53
|
+
#
|
|
54
|
+
# On aarch64-linux, the two names collide. The gem needs duckling.so,
|
|
55
|
+
# which is the name the local build makes. So the packaging step waits
|
|
56
|
+
# for the local build. That build compiles code for aarch64, but links
|
|
57
|
+
# the code with the host's plain gcc. rake-compiler picked this gcc for
|
|
58
|
+
# the host platform. The link step then fails, and stops the build.
|
|
59
|
+
#
|
|
60
|
+
# extconf.rb runs in its own subprocess. ENV changes made there do not
|
|
61
|
+
# reach the parent process, and the parent process runs `make`.
|
|
62
|
+
#
|
|
63
|
+
# So this fix changes ENV here, in the Rakefile.
|
|
64
|
+
# The fix adds a prerequisite task to the local build's Makefile task.
|
|
65
|
+
# This prerequisite task sets the correct host target. It runs before
|
|
66
|
+
# the Makefile task, and before `make` runs later.
|
|
67
|
+
if (ruby_target = ENV["RUBY_TARGET"]) && ruby_target != RUBY_PLATFORM
|
|
68
|
+
local_makefile = "tmp/#{RUBY_PLATFORM}/duckling/#{RUBY_VERSION}/Makefile"
|
|
69
|
+
|
|
70
|
+
# CARGO_BUILD_TARGET must hold a real triple before RUST_TARGET goes
|
|
71
|
+
# away. Clearing RUST_TARGET alone lets rb_sys fall back to the target
|
|
72
|
+
# baked into the container's $CARGO_HOME/config.toml, which is the
|
|
73
|
+
# cross-compile target again. So a host triple that cannot be read is a
|
|
74
|
+
# hard error.
|
|
75
|
+
#
|
|
76
|
+
# Both variables belong to the whole rake process, and the cross build
|
|
77
|
+
# reads them too. This is safe only because rake generates the cross
|
|
78
|
+
# Makefile, which bakes its own --target in, before packaging reaches
|
|
79
|
+
# the local pass. Nothing in rake states that order. If it ever
|
|
80
|
+
# inverted, the cross build would compile for the host and produce a
|
|
81
|
+
# correctly *named* binary for the wrong architecture — which is what
|
|
82
|
+
# `file(1)` on every binary in test/gem/packaged_gem_test.rb catches.
|
|
83
|
+
task :fix_local_pass_cargo_target do
|
|
84
|
+
rustc_version_info = begin
|
|
85
|
+
`rustc -vV`
|
|
86
|
+
rescue Errno::ENOENT
|
|
87
|
+
raise "Cannot run `rustc -vV` to find the host target triple. rustc must be on PATH."
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
host_target = rustc_version_info[/^host: (\S+)$/, 1]
|
|
91
|
+
raise "`rustc -vV` printed no `host:` line:\n#{rustc_version_info}" unless host_target
|
|
92
|
+
|
|
93
|
+
ENV["CARGO_BUILD_TARGET"] = host_target
|
|
94
|
+
ENV.delete("RUST_TARGET")
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
# local_makefile reconstructs a path rake-compiler builds from its own
|
|
98
|
+
# internals (tmp_dir, extension name, the local pass's Ruby version).
|
|
99
|
+
# A gem upgrade can change any of them. Say so here, because the
|
|
100
|
+
# alternative is a silent no-op and a link failure deep in a container.
|
|
101
|
+
unless Rake::Task.task_defined?(local_makefile)
|
|
102
|
+
raise "Expected rake-compiler to define a Makefile task at #{local_makefile}. " \
|
|
103
|
+
"The local-pass Cargo target override needs that exact task name — check " \
|
|
104
|
+
"define_compile_tasks in rake-compiler's extensiontask.rb for the current path."
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
Rake::Task[local_makefile].enhance([:fix_local_pass_cargo_target])
|
|
108
|
+
end
|
|
109
|
+
|
|
110
|
+
task :dev do
|
|
111
|
+
ENV["RB_SYS_CARGO_PROFILE"] = "dev"
|
|
112
|
+
end
|
|
113
|
+
|
|
114
|
+
# rb-sys-dock mounts the working directory into the build container with
|
|
115
|
+
# `-v $(pwd):$(pwd)` and nothing else. In a git worktree — which is how
|
|
116
|
+
# bin/worktree sets up every branch here — .git is a *file* naming a path under
|
|
117
|
+
# the main checkout's .git/worktrees/, which that mount does not cover. Every
|
|
118
|
+
# git command inside the container then fails, including the `git ls-files` in
|
|
119
|
+
# duckling.gemspec, whose file list comes back empty. The gem that comes out
|
|
120
|
+
# holds the compiled binaries and no Ruby at all.
|
|
121
|
+
#
|
|
122
|
+
# So build from a throwaway plain clone instead, whose .git is a real
|
|
123
|
+
# directory under the mount. Two consequences: the gem carries *committed*
|
|
124
|
+
# state (uncommitted changes are excluded), and the clone gets its own Cargo
|
|
125
|
+
# target directory separate from this checkout's.
|
|
126
|
+
CLONE_DIR = "tmp/native_gem_clone"
|
|
127
|
+
|
|
128
|
+
def build_native_gem(platform)
|
|
129
|
+
# bundler exports BUNDLE_GEMFILE pointing at this checkout, and rb-sys-dock
|
|
130
|
+
# mounts $(pwd) — the two have to move together or the container gets one
|
|
131
|
+
# directory's Gemfile and another's source.
|
|
132
|
+
Bundler.with_unbundled_env do
|
|
133
|
+
sh "bundle", "install"
|
|
134
|
+
sh "bundle", "exec", "rb-sys-dock", "--platform", platform,
|
|
135
|
+
"--ruby-versions", CrossTargets::RUBY_ABIS.join(","), "--build"
|
|
136
|
+
end
|
|
137
|
+
end
|
|
138
|
+
|
|
139
|
+
desc "Cross-compile the native extension for a given platform via rb-sys-dock (e.g. `rake 'native_gem[x86_64-linux]'`)"
|
|
140
|
+
task :native_gem, [:platform] do |_t, platform:|
|
|
141
|
+
next build_native_gem(platform) unless File.file?(".git")
|
|
142
|
+
|
|
143
|
+
head = `git rev-parse HEAD`.strip
|
|
144
|
+
raise "Could not read HEAD to pin the build clone." if head.empty?
|
|
145
|
+
|
|
146
|
+
unless `git status --porcelain`.empty?
|
|
147
|
+
warn "native_gem: building #{head[0, 7]} from a clone — uncommitted changes are not in this gem."
|
|
148
|
+
end
|
|
149
|
+
|
|
150
|
+
rm_rf CLONE_DIR
|
|
151
|
+
begin
|
|
152
|
+
sh "git", "clone", "--local", "--no-checkout", Dir.pwd, CLONE_DIR
|
|
153
|
+
sh "git", "-C", CLONE_DIR, "checkout", "--detach", head
|
|
154
|
+
|
|
155
|
+
Dir.chdir(CLONE_DIR) { build_native_gem(platform) }
|
|
156
|
+
|
|
157
|
+
mkdir_p "pkg"
|
|
158
|
+
cp FileList["#{CLONE_DIR}/pkg/*.gem"], "pkg"
|
|
159
|
+
ensure
|
|
160
|
+
rm_rf CLONE_DIR
|
|
161
|
+
end
|
|
162
|
+
end
|
|
163
|
+
|
|
164
|
+
task :benchmark_env do
|
|
165
|
+
# Force a realistic release-profile build regardless of .env.local's
|
|
166
|
+
# RB_SYS_CARGO_PROFILE=dev (local dev checkouts only; CI never has it).
|
|
167
|
+
# Must reenable :compile in case it already ran earlier in this same
|
|
168
|
+
# rake process, so it recompiles under the forced profile. A stale
|
|
169
|
+
# dev-profile build would be reused otherwise.
|
|
170
|
+
ENV.delete("RB_SYS_CARGO_PROFILE")
|
|
171
|
+
Rake::Task[:compile].reenable
|
|
172
|
+
end
|
|
173
|
+
|
|
174
|
+
desc "Run the benchmark-ips suite (console output only, no file writes)"
|
|
175
|
+
task benchmark: [:benchmark_env, :compile] do
|
|
176
|
+
ruby "-Ilib", "benchmark/parse_benchmark.rb"
|
|
177
|
+
end
|
|
178
|
+
|
|
179
|
+
namespace :benchmark do
|
|
180
|
+
desc "Run benchmarks, write docs/benchmarks/<environment>/<version>.json, regenerate docs/benchmarks/README.md"
|
|
181
|
+
task record: [:benchmark_env, :compile] do
|
|
182
|
+
ruby "-Ilib", "benchmark/report.rb"
|
|
183
|
+
end
|
|
184
|
+
|
|
185
|
+
desc "Run :record on a fresh branch off origin/main, then commit/push and open+auto-merge a PR via gh"
|
|
186
|
+
task record_pr: ["release:guard_clean"] do
|
|
187
|
+
# Explicit bash: Rake's default `sh -c` is dash on Debian/Ubuntu
|
|
188
|
+
# runners, and dash's `set` doesn't support the `-o pipefail` flag below.
|
|
189
|
+
sh("bash", "-c", <<~SH)
|
|
190
|
+
set -euo pipefail
|
|
191
|
+
original_ref="$(git symbolic-ref -q --short HEAD || git rev-parse HEAD)"
|
|
192
|
+
git fetch origin main
|
|
193
|
+
git checkout -b "benchmark/pending-$(date +%s)" origin/main
|
|
194
|
+
|
|
195
|
+
bundle exec rake benchmark:record
|
|
196
|
+
|
|
197
|
+
version="$(ruby -Ilib -e 'require "duckling"; puts Duckling::VERSION')"
|
|
198
|
+
environment="$(ruby -Ilib -e 'require_relative "benchmark/report"; puts DucklingBenchmark::Report::ENVIRONMENT')"
|
|
199
|
+
branch="benchmark/${environment}/${version}-$(date +%s)"
|
|
200
|
+
git branch -m "$branch"
|
|
201
|
+
|
|
202
|
+
git add docs/benchmarks
|
|
203
|
+
git commit -m "Record ${environment} benchmark results for ${version}"
|
|
204
|
+
git push origin "$branch"
|
|
205
|
+
gh pr create --base main --head "$branch" \\
|
|
206
|
+
--title "Benchmark results (${environment}, ${version})" \\
|
|
207
|
+
--body "Automated benchmark recording from ${environment}."
|
|
208
|
+
gh pr merge "$branch" --auto --squash
|
|
209
|
+
|
|
210
|
+
git checkout "$original_ref"
|
|
211
|
+
git branch -D "$branch"
|
|
212
|
+
SH
|
|
213
|
+
end
|
|
214
|
+
end
|
|
215
|
+
|
|
216
|
+
# The default suite exercises the extension compiled in this checkout.
|
|
217
|
+
# Three test subtrees run outside it:
|
|
218
|
+
# - test/gem/ exercises a *built* or *installed* gem instead — it needs one
|
|
219
|
+
# handed to it, and the installed suite must not see this checkout's lib/.
|
|
220
|
+
# - test/capabilities/ is loaded by test_helper itself, gated on the tz probes.
|
|
221
|
+
# - test/environments/ holds contracts invoked directly by their CI step.
|
|
222
|
+
# See docs/tz-database-axis.md.
|
|
223
|
+
Minitest::TestTask.create do |t|
|
|
224
|
+
t.test_globs = FileList["test/**/*_test.rb"].exclude("test/gem/**/*", "test/capabilities/**/*", "test/environments/**/*")
|
|
225
|
+
end
|
|
226
|
+
|
|
227
|
+
# Minitest::TestTask has no built-in way to declare a task dependency, and
|
|
228
|
+
# `task default: %i[standard compile test]`'s array ordering only protects
|
|
229
|
+
# `bundle exec rake` itself — `bundle exec rake test` run directly has no
|
|
230
|
+
# guarantee `compile` ran first, which would surface as a confusing
|
|
231
|
+
# LoadError/stale-behavior failure unrelated to the code under test.
|
|
232
|
+
task test: %i[compile]
|
|
12
233
|
|
|
13
234
|
require "standard/rake"
|
|
14
235
|
|
|
15
236
|
task default: %i[standard compile test]
|
|
237
|
+
|
|
238
|
+
# bundler/gem_tasks's default `release` task builds and pushes the .gem
|
|
239
|
+
# itself, which would race the tag-triggered CI pipeline in
|
|
240
|
+
# .github/workflows/release.yml that already does the actual build and
|
|
241
|
+
# publish once a vX.Y.Z tag lands. Narrow `release` to just tagging.
|
|
242
|
+
Rake::Task["release"].clear
|
|
243
|
+
task release: ["release:guard_clean", "release:source_control_push"]
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
# Roadmap — Issue #1: Ship duckling gem (time extraction via Magnus + wafer-inc-duckling)
|
|
2
|
+
|
|
3
|
+
**0.2.0 has shipped.** The original three-plan implementation sequence (native
|
|
4
|
+
extension → Ruby API → test suite/CI) fully executed — see
|
|
5
|
+
[the wiki's stale plans](https://github.com/cpb/duckling/wiki/plans-stale) for
|
|
6
|
+
that historical record. All research and design-exploration docs that grounded these
|
|
7
|
+
decisions have moved to [the project wiki](https://github.com/cpb/duckling/wiki); this
|
|
8
|
+
file is the only planning document that stays in the repo.
|
|
9
|
+
|
|
10
|
+
This document is the live plan: **what's left, tracked as GitHub issues**, not prose
|
|
11
|
+
duplicated here. Nothing below needs to happen for the gem to work today — these are
|
|
12
|
+
0.2.x-and-beyond follow-ups raised during PR #3 review, while re-verifying the stale
|
|
13
|
+
plans against what actually shipped, and during issue #57's research into the
|
|
14
|
+
async-reactor-blocking fix.
|
|
15
|
+
|
|
16
|
+
If you're new here, read this file top-to-bottom once; after that, treat it as a
|
|
17
|
+
living index and jump straight to a section via the table of contents below.
|
|
18
|
+
|
|
19
|
+
## Table of contents
|
|
20
|
+
|
|
21
|
+
| Section | Covers |
|
|
22
|
+
|---|---|
|
|
23
|
+
| [Environment & tooling alignment](#environment--tooling-alignment) | Keeping local dev, Claude Code Web, and CI on the same Ruby/Rust versions |
|
|
24
|
+
| [Developer workflow (Rakefile)](#developer-workflow-rakefile) | Rake task dependency ordering |
|
|
25
|
+
| [API design exploration (post-0.2.0 direction)](#api-design-exploration-post-020-direction) | Whether 0.2.0's manual Magnus/Hash API is the final shape |
|
|
26
|
+
| [Test coverage](#test-coverage) | Extending and auditing the test corpus |
|
|
27
|
+
| [Performance & concurrency](#performance--concurrency) | Benchmarking, and unblocking `Duckling.parse` under Fiber-scheduled reactors |
|
|
28
|
+
| [Settled Decisions (0.2.0, verified shipped)](#settled-decisions-020-verified-shipped) | Choices already made and shipped — no action needed |
|
|
29
|
+
| [Further reading](#further-reading) | Where the research that grounded all of the above lives |
|
|
30
|
+
|
|
31
|
+
## Environment & tooling alignment
|
|
32
|
+
|
|
33
|
+
Keep local dev, Claude Code Web, and CI targeting the same Ruby/Rust versions so nothing
|
|
34
|
+
needs manual installation in any of them.
|
|
35
|
+
|
|
36
|
+
- ~~[#26](https://github.com/cpb/duckling/issues/26)~~ `bin/setup` Brewfile step for Rust — **shipped**
|
|
37
|
+
- ~~[#27](https://github.com/cpb/duckling/issues/27)~~ `bin/claude-code-web-setup` just-in-time deps — **shipped**
|
|
38
|
+
- ~~[#28](https://github.com/cpb/duckling/issues/28)~~ Pin CI's Rust toolchain to Claude Code Web's version — **shipped**
|
|
39
|
+
|
|
40
|
+
| Issue | What |
|
|
41
|
+
|-------|------|
|
|
42
|
+
| [#29](https://github.com/cpb/duckling/issues/29) | Expand the CI matrix to Ruby 3.4 / latest Ruby / latest Rust (forward-compat signal) |
|
|
43
|
+
| [#43](https://github.com/cpb/duckling/issues/43) | Publish precompiled binary gems for `x86_64-darwin-24` and `x86_64-linux` |
|
|
44
|
+
|
|
45
|
+
## Developer workflow (Rakefile)
|
|
46
|
+
|
|
47
|
+
- ~~[#30](https://github.com/cpb/duckling/issues/30)~~ `:dev` Rake task (`RB_SYS_CARGO_PROFILE=dev`) — **shipped**
|
|
48
|
+
|
|
49
|
+
| Issue | What |
|
|
50
|
+
|-------|------|
|
|
51
|
+
| [#31](https://github.com/cpb/duckling/issues/31) | Explore making `test` explicitly depend on `:compile`, instead of relying on `default` task array ordering |
|
|
52
|
+
|
|
53
|
+
## API design exploration (post-0.2.0 direction)
|
|
54
|
+
|
|
55
|
+
The 0.2.0 API (manual Magnus hash mapping, matching pyduckling's Hash-based shape) is
|
|
56
|
+
not necessarily the final shape — see "Option D" in
|
|
57
|
+
[research-type-mapping-strategy-serialization-options](https://github.com/cpb/duckling/wiki/research-type-mapping-strategy-serialization-options)
|
|
58
|
+
on the wiki.
|
|
59
|
+
|
|
60
|
+
| Issue | What |
|
|
61
|
+
|-------|------|
|
|
62
|
+
| [#32](https://github.com/cpb/duckling/issues/32) | Explore serde_magnus (symbol keys) + Ruby pattern-matching `Data` factories as a Hash-free API |
|
|
63
|
+
| [#33](https://github.com/cpb/duckling/issues/33) | v0.3.0: handle Naive time values the Rails ActiveSupport way (resolve against reference zone) |
|
|
64
|
+
| [#45](https://github.com/cpb/duckling/issues/45) | `reference_time:` — accept a Ruby `Time` object to preserve UTC offset |
|
|
65
|
+
| [#46](https://github.com/cpb/duckling/issues/46) | Implement the remaining 13 `DimensionValue` variants beyond `Time` |
|
|
66
|
+
| [#47](https://github.com/cpb/duckling/issues/47) | Explore upstreaming serde container attributes to wafer-inc/duckling |
|
|
67
|
+
|
|
68
|
+
## Test coverage
|
|
69
|
+
|
|
70
|
+
| Issue | What |
|
|
71
|
+
|-------|------|
|
|
72
|
+
| [#34](https://github.com/cpb/duckling/issues/34) | Implement the extended test corpus designed in [research-test-coverage-ruby-test-design](https://github.com/cpb/duckling/wiki/research-test-coverage-ruby-test-design) on the wiki (`test-first`) |
|
|
73
|
+
| [#35](https://github.com/cpb/duckling/issues/35) | Audit wafer-inc-duckling's test coverage against pyduckling and upstream Haskell duckling |
|
|
74
|
+
|
|
75
|
+
## Performance & concurrency
|
|
76
|
+
|
|
77
|
+
- ~~[#36](https://github.com/cpb/duckling/issues/36)~~ `benchmark-ips` suite with automated README/CHANGELOG reporting — **shipped** (`benchmark/parse_benchmark.rb`, `bin/benchmark record-pr`; see `docs/benchmarks/README.md`)
|
|
78
|
+
|
|
79
|
+
| Issue | What |
|
|
80
|
+
|-------|------|
|
|
81
|
+
| [#38](https://github.com/cpb/duckling/issues/38) | Test-drive the Falcon Fiber-blocking claim in [research-ffi-risks](https://github.com/cpb/duckling/wiki/research-ffi-risks) on the wiki (`test-first`) — hill test drafted in [PR #50](https://github.com/cpb/duckling/pull/50), pending human review |
|
|
82
|
+
| [#64](https://github.com/cpb/duckling/issues/64) | Implement thread-per-call GVL release (`rb_thread_call_without_gvl` + a spawned background `Thread`) to make `test/falcon_fiber_blocking_test.rb` pass once #38's hill lands — see [research-async-reactor-blocking](https://github.com/cpb/duckling/wiki/research-async-reactor-blocking) on the wiki for why a bare GVL release isn't enough. Includes splitting `Duckling::Native.parse` out from `Duckling.parse` as a benchmarking seam, and a with-thread-vs-without-thread benchmark comparison. |
|
|
83
|
+
|
|
84
|
+
**Deferred follow-ups from #57's research** (explicitly out of scope for #64 itself — revisit only if their trigger condition below actually occurs):
|
|
85
|
+
|
|
86
|
+
- **Persistent worker-pool dispatch** instead of thread-per-call — only worth it if a production workload dominated by many back-to-back `empty`/`no_match`-shaped calls makes the ~70µs/call thread-spawn floor a measurable fraction of total latency. See [research-concurrency-alternatives-comparison-worker-pool](https://github.com/cpb/duckling/wiki/research-concurrency-alternatives-comparison-worker-pool) — a single-worker pool would serialize concurrent callers despite the wrapped crate supporting true concurrency, so this is a real regression risk, not a free win; a *bounded* pool (N > 1) was flagged as uninvestigated if this is ever picked up.
|
|
87
|
+
- **Conditional thread-spawn based on `Fiber.current_scheduler`** (skip the `Thread.new` when no reactor is present, e.g. a plain script or non-Fiber-scheduled Puma worker) — the detection signal was analyzed as sound (correctly distinguishes "no reactor" from "reactor present but current Fiber is blocking," both cases where spawning a thread buys nothing) but was never exercised in a test. Deferred because #64 (and #57 more broadly) has no throughput-optimization goal — adding a conditional branch here is itself a latency optimization for the non-reactor case, the same category of complexity rejected for the worker-pool alternative. Revisit if the ~70µs/call floor is ever shown to matter for a non-reactor caller in practice.
|
|
88
|
+
- **Ruby version floor** (`>= 3.2.0` today) — [#77](https://github.com/cpb/duckling/issues/77) spiked whether calling `rb_nogvl` directly with `RB_NOGVL_OFFLOAD_SAFE` (instead of `rb_thread_call_without_gvl`, which #64 uses and which always passes `flags: 0`) lets `Fiber::Scheduler#blocking_operation_wait` auto-offload `duckling::parse`, obviating #64's Thread wrapper on that floor. **Verdict: CONFIRMED** on Ruby 3.4+ — see the wiki's `research-rb-nogvl-offload-safe-spike` for the full methodology and data. Important caveat found along the way: `io-event`'s automatic `WorkerPool` support (what lets the real `async`/Falcon gem exercise this hook without a hand-rolled scheduler) requires `rb_fiber_scheduler_blocking_operation_extract`, which only ships in Ruby 4.0+ — so `async`-based callers won't see any benefit until they're on Ruby 4.0, even though the underlying mechanism works on 3.4. Not proposing a floor bump for #64's sake either way; a follow-up implementation issue to actually add the 3.4+-gated dispatch path is recommended but was deferred to #77's PR review rather than opened by the spike itself.
|
|
89
|
+
|
|
90
|
+
## Settled Decisions (0.2.0, verified shipped)
|
|
91
|
+
|
|
92
|
+
- **[duckling](https://github.com/wafer-inc/duckling) on crates.io** — Published as `duckling = "0.4"`.
|
|
93
|
+
- **Symbol keys and Symbol values throughout** — `:body`, `:dim`, `:value`, `:type`, `:grain`, etc. Settled by the hill tests in PR #2, confirmed shipped in `test/duckling_test.rb` on `main`.
|
|
94
|
+
- **NaiveDateTime → bare ISO8601 (no offset)** — Option N1. Shipped as-is; Option N2 (ActiveSupport-style zone resolution) is tracked for 0.3.0 as [#33](https://github.com/cpb/duckling/issues/33).
|
|
95
|
+
- **Manual Magnus mapping, not serde_magnus** — shipped as Option B; `magnus = "0.8"` (not `"0.9"`, which was never published to crates.io).
|
|
96
|
+
- **Source gem, not pre-compiled binaries** — shipped this way for 0.2.0; [#43](https://github.com/cpb/duckling/issues/43) tracks adding pre-compiled binary gems.
|
|
97
|
+
|
|
98
|
+
## Further reading
|
|
99
|
+
|
|
100
|
+
The full research that grounded these decisions — wafer-inc-duckling's API surface,
|
|
101
|
+
Magnus/rb-sys build wiring, type-mapping strategy options, test corpus design,
|
|
102
|
+
empirical FFI risk analysis (GVL blocking, panic safety, GC pressure, day-of-week
|
|
103
|
+
validation gap), and issue #57's async-reactor-blocking investigation (GVL-release
|
|
104
|
+
mechanics, the wrapped crate's thread/panic-safety, dispatch-strategy comparison, and
|
|
105
|
+
the empirical spike proving GVL-release-alone doesn't work) — lives on
|
|
106
|
+
[the project wiki](https://github.com/cpb/duckling/wiki), not in this repo.
|