nukadoko 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +15 -0
- data/LICENSE +21 -0
- data/README.ja.md +71 -0
- data/README.md +66 -0
- data/exe/nuka +7 -0
- data/lib/nukadoko/check.rb +173 -0
- data/lib/nukadoko/claims.rb +110 -0
- data/lib/nukadoko/cli.rb +76 -0
- data/lib/nukadoko/command.rb +63 -0
- data/lib/nukadoko/definition.rb +68 -0
- data/lib/nukadoko/features.rb +50 -0
- data/lib/nukadoko/fixtures.rb +80 -0
- data/lib/nukadoko/parts.rb +68 -0
- data/lib/nukadoko/pattern.rb +29 -0
- data/lib/nukadoko/poll.rb +75 -0
- data/lib/nukadoko/program.rb +70 -0
- data/lib/nukadoko/project_fixtures.rb +60 -0
- data/lib/nukadoko/run.rb +224 -0
- data/lib/nukadoko/schema.rb +162 -0
- data/lib/nukadoko/signature.rb +30 -0
- data/lib/nukadoko/step.rb +98 -0
- data/lib/nukadoko/table.rb +47 -0
- data/lib/nukadoko/version.rb +5 -0
- data/lib/nukadoko/vocabulary.rb +56 -0
- data/lib/nukadoko.rb +28 -0
- data/sig/nukadoko.rbs +60 -0
- metadata +107 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: b6d047ee71c9d370c302adba5f7746fc297ddb145f34a7edb4aadf19d6b9174c
|
|
4
|
+
data.tar.gz: 9498233a5e767889d79c193ac243f0e4a14fea417bca0ace600b7391ed077916
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 823e46b1d92101a548e63b42ab808d0ae3739d53179c037ce6af1ecdca5622fde9694b34b6080f646e8942c17f793ad66ed1df4a81a3cff4e816ee905a6854c9
|
|
7
|
+
data.tar.gz: 0c768c8cd2e6ef4d3eeb28cbc02cb58be4e34b038996429687630261490db2ce95a6428ff6f243d94cc43c6f4bea4504c7710f4ac37bf46e1e8a05ea28c23169
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
Notable changes to the `nukadoko` gem, the Ruby engine. The npm package has
|
|
4
|
+
its own changelog, `../CHANGELOG.md`, and its own versions. Each release's
|
|
5
|
+
notes come from the section below whose heading starts with `## [<version>]`.
|
|
6
|
+
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [0.1.0] - 2026-11-14
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- The first Ruby engine: steps written as Ruby classes with typed `args` and
|
|
14
|
+
`returns`, `nuka check`, `nuka run`, `nuka steps`, `nuka describe`, and
|
|
15
|
+
`nuka sig` for Steep. It passes every case under `conformance/`.
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 meganemura
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.ja.md
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# nukadoko の Ruby 版
|
|
2
|
+
|
|
3
|
+
[nukadoko](../README.ja.md) の Ruby のエンジンです。
|
|
4
|
+
Gherkin のステップに型付きの契約を付け、シナリオのまちがいを実行前に見つけます。
|
|
5
|
+
TypeScript 版と同じ feature ファイルを読み、同じ判定を返す必要があります。
|
|
6
|
+
[`../conformance/`](../conformance/README.md) のケースがそれを確かめます。
|
|
7
|
+
|
|
8
|
+
状態: 初期の核です。
|
|
9
|
+
`conformance/` の 10 のケースを通します。
|
|
10
|
+
HTTP のステップ、ブラウザのステップ、コマンドラインのステップ、プロジェクトが定義するフィクスチャ、parts でできたステップ、poll する Then ステップ、`from`、docstring、Scenario Outline、条件つきの環境、control と proof run、判定表です。
|
|
11
|
+
`--base`、`nuka check` の指摘の多くなど、TypeScript 版の機能の多くはまだありません。
|
|
12
|
+
|
|
13
|
+
## インストール
|
|
14
|
+
|
|
15
|
+
テストするプロジェクトの Gemfile に足します。
|
|
16
|
+
Ruby 3.4 以上が要ります。
|
|
17
|
+
|
|
18
|
+
```ruby
|
|
19
|
+
gem "nukadoko", group: :test
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
## ステップ
|
|
23
|
+
|
|
24
|
+
`features/steps/` の下に 1 ファイル 1 ステップで書きます。
|
|
25
|
+
ファイル名がステップ名です(`add_todo.rb` は `add-todo`)。
|
|
26
|
+
|
|
27
|
+
```ruby
|
|
28
|
+
class AddTodo < Nukadoko::Step
|
|
29
|
+
pattern "a todo titled {title:string} is added"
|
|
30
|
+
args { string :title, "The title of the todo to create" }
|
|
31
|
+
returns do
|
|
32
|
+
string :id
|
|
33
|
+
string :title
|
|
34
|
+
boolean :done
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
def run(request:, args:)
|
|
38
|
+
request.post("/todos", data: { title: args.title }).json
|
|
39
|
+
end
|
|
40
|
+
end
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
- nil を許すフィールドには `null: true` を付けます。`enum :outcome, %w[accepted refused]` はその語だけを許し、Steep はリテラルの和として扱います。
|
|
44
|
+
- パターンの取り出し部分にはすべて名前を付けます。名前は `args` のキーです。
|
|
45
|
+
- `run`(Then ステップでは `observe`)のキーワード引数が、ステップが使うフィクスチャです。`nuka check` は本体を呼ばずに `Method#parameters` で読みます。
|
|
46
|
+
- Then ステップは `claims` と、世界を読む `observe` を書きます。claim の判定は nukadoko が行います。claim が比べる args のキーは `observe` に渡されません。
|
|
47
|
+
|
|
48
|
+
## コマンド
|
|
49
|
+
|
|
50
|
+
- `nuka check [--json]` は、実行前にまちがいを報告します。
|
|
51
|
+
- `nuka run [--json] [--no-check]` は、プログラムを起動してすべてのシナリオを実行し、ステップごとの記録を `.nukadoko/runs/` に書きます。
|
|
52
|
+
- ブラウザの起動設定は `nukadoko.config.rb` に `Nukadoko.config(browser: { executablePath: "..." })` と書きます。`page` を使うステップは、シナリオごとに自分のブラウザコンテキストを持ちます。ステップの記録には、ページが送ったリクエストとステップのトレースが残ります。
|
|
53
|
+
- `scratch` を使うステップは `scratch.dir` で新しいディレクトリを得ます。シナリオが終わると消えます。`Nukadoko.run_command(command, args, cwd:)` はプログラムを実行し、`{ exitCode:, stdout:, stderr: }` を返します。ステップの記録には実行したプロセスが残ります。
|
|
54
|
+
- プロジェクトのフィクスチャは `nukadoko.config.rb` に書きます。ブロックは値を `use` に渡し、その後ろが後片付けです。例: `Nukadoko.fixture(:store) { |use| use.(Store.new) }`。`scope: :process` にすると、環境ごとの実行全体で 1 つの値を使います。
|
|
55
|
+
- `parts [CreateTodo, CompleteById]` で、ステップが呼べるステップを宣言します。`call.(CreateTodo, title: args.title)` で 1 つを実行し、args と返り値を確かめます。パターンのない part も書けます。呼び出しは、呼んだステップの記録の `calls` に残ります。`nuka check` は、`parts` の循環と、`mutates false` なのに `mutates true` の part を持つステップを報告します。
|
|
56
|
+
- `poll timeout_ms: 3000` は、すべての claim が 2 回続けて成り立つか時間切れになるまで、100 ミリ秒おきに `observe` を読み直します。`settled:` には、一度なったら変わらない状態を書きます。2 回続けてその状態なら、壊れた claim を待つのをやめます。読んだ結果はすべてステップの記録に残ります。
|
|
57
|
+
- `nuka steps [--json]` は語彙の一覧を出します。`nuka describe <step>` は 1 つのステップを、args と返り値の JSON Schema とともに出します。項目は TypeScript 版と同じです。
|
|
58
|
+
- 契約が宣言していない返り値のキーは、zod のオブジェクトと同じく捨てられます。ステップの記録の `returns_dropped` に、捨てたキーが残ります。
|
|
59
|
+
- `nuka sig` は `sig/nukadoko/steps.rbs` を書きます。Steep を使うと、宣言していない args のキーを読む本体や、形のちがう値を返す本体がエラーになります。
|
|
60
|
+
|
|
61
|
+
テストするプログラムは `nukadoko.program.rb` に書きます。
|
|
62
|
+
|
|
63
|
+
```ruby
|
|
64
|
+
Nukadoko.program serve: { command: "bin/rails server -p {port}", ready: "/up" }
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## 開発
|
|
68
|
+
|
|
69
|
+
- このディレクトリで `bundle install` を実行します。
|
|
70
|
+
- `bundle exec ruby ../conformance/run-rb.rb` で、すべての検査ケースをこのエンジンで走らせます。
|
|
71
|
+
- `bundle exec bin/test` で単体テストと、`test/hegel/` の Hegel の性質テストを走らせます。gem には Ruby 3.4 以上が要ります。
|
data/README.md
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# nukadoko for Ruby
|
|
2
|
+
|
|
3
|
+
The Ruby engine of [nukadoko](../README.md): typed step contracts for Gherkin, written in Ruby, so a scenario can be told wrong before it runs.
|
|
4
|
+
It reads the same feature files as the TypeScript engine and must judge them the same way; the cases in [`../conformance/`](../conformance/README.md) hold it to that.
|
|
5
|
+
|
|
6
|
+
Status: an early core. It passes the ten cases in `conformance/`: HTTP steps, browser steps, command-line steps, fixtures a project defines, steps made of parts, polling Then steps, `from`, docstrings, Scenario Outlines, environments with conditions, controls with proof runs, and the verdict table. Much of what the TypeScript engine does is not here yet, such as `--base`, and most of what `nuka check` reports.
|
|
7
|
+
|
|
8
|
+
## Install
|
|
9
|
+
|
|
10
|
+
Add it to the Gemfile of the project under test. It needs Ruby 3.4 or later.
|
|
11
|
+
|
|
12
|
+
```ruby
|
|
13
|
+
gem "nukadoko", group: :test
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
## A step
|
|
17
|
+
|
|
18
|
+
One step per file under `features/steps/`. The file name is the step name (`add_todo.rb` is `add-todo`).
|
|
19
|
+
|
|
20
|
+
```ruby
|
|
21
|
+
class AddTodo < Nukadoko::Step
|
|
22
|
+
pattern "a todo titled {title:string} is added"
|
|
23
|
+
args { string :title, "The title of the todo to create" }
|
|
24
|
+
returns do
|
|
25
|
+
string :id
|
|
26
|
+
string :title
|
|
27
|
+
boolean :done
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def run(request:, args:)
|
|
31
|
+
request.post("/todos", data: { title: args.title }).json
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
- A field takes `null: true` when nil is allowed. `enum :outcome, %w[accepted refused]` allows only those words, and Steep sees them as a union of literals.
|
|
37
|
+
- Every capture in the pattern is named, and each name is a key of `args`.
|
|
38
|
+
- The keyword arguments of `run` (or `observe`, in a Then step) name the fixtures the step needs. `nuka check` reads them with `Method#parameters`, without calling the body.
|
|
39
|
+
- A Then step declares `claims` and an `observe` that reads the world. nukadoko judges the claims. `observe` does not receive an args key that a claim compares with.
|
|
40
|
+
|
|
41
|
+
## Commands
|
|
42
|
+
|
|
43
|
+
- `nuka check [--json]` reports what is wrong before anything runs.
|
|
44
|
+
- `nuka run [--json] [--no-check]` starts the program, runs every scenario, and writes a record per step under `.nukadoko/runs/`.
|
|
45
|
+
- Browser launch options go in `nukadoko.config.rb`, as `Nukadoko.config(browser: { executablePath: "..." })`. A step that names `page` gets a browser context of its own for each scenario; each step record holds the requests the page sent and a trace of the step.
|
|
46
|
+
- A step that names `scratch` gets `scratch.dir`, a fresh directory that is removed when the scenario ends. `Nukadoko.run_command(command, args, cwd:)` runs a program and returns `{ exitCode:, stdout:, stderr: }`; the step record lists each run.
|
|
47
|
+
- A project fixture goes in `nukadoko.config.rb`. The block calls `use` with the value, and what follows is the teardown: `Nukadoko.fixture(:store) { |use| use.(Store.new) }`. `scope: :process` keeps one value for the whole run of an environment.
|
|
48
|
+
- `parts [CreateTodo, CompleteById]` declares the steps a step may run, and `call.(CreateTodo, title: args.title)` runs one, with its args and returns checked. A part needs no pattern. Each call is kept under `calls` in the calling step's record. `nuka check` reports a cycle in `parts` and a `mutates false` step whose part declares `mutates true`.
|
|
49
|
+
- `poll timeout_ms: 3000` reads `observe` again, 100 ms apart, until every claim holds on two readings in a row or the time runs out. `settled:` names a state that cannot change once reached, so two settled readings in a row stop the wait for a claim that broke. Every reading is kept in the step record.
|
|
50
|
+
- `nuka steps [--json]` lists the vocabulary, and `nuka describe <step>` prints one step with the JSON Schema of its args and returns, in the fields the TypeScript engine prints.
|
|
51
|
+
- A returned key the contract does not declare is dropped, as a zod object drops it, and the step record names it under `returns_dropped`.
|
|
52
|
+
- `nuka sig` writes `sig/nukadoko/steps.rbs`. With Steep, a body that reads an undeclared args key or returns the wrong shape is an error.
|
|
53
|
+
|
|
54
|
+
The program under test is described in `nukadoko.program.rb`:
|
|
55
|
+
|
|
56
|
+
```ruby
|
|
57
|
+
Nukadoko.program serve: { command: "bin/rails server -p {port}", ready: "/up" }
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
## Development
|
|
61
|
+
|
|
62
|
+
- `bundle install` in this directory.
|
|
63
|
+
- `bundle exec ruby ../conformance/run-rb.rb` runs every conformance case under this engine.
|
|
64
|
+
- `bundle exec bin/test` runs the unit tests and the Hegel property tests under `test/hegel/`. The gem needs Ruby 3.4 or later.
|
|
65
|
+
|
|
66
|
+
[日本語](README.ja.md)
|
data/exe/nuka
ADDED
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# What `check` can say without running anything: every finding follows from
|
|
4
|
+
# the step classes, their patterns and schemas, `Method#parameters`, and the
|
|
5
|
+
# feature files. The codes are the TypeScript engine's codes, so one set of
|
|
6
|
+
# conformance cases judges both engines.
|
|
7
|
+
module Nukadoko
|
|
8
|
+
module Engine
|
|
9
|
+
Finding = Data.define(:severity, :code, :file, :line, :step, :message, :fix) do
|
|
10
|
+
def self.about_step(severity, code, step, message, fix)
|
|
11
|
+
new(severity, code, step.file, step.line, step.step_name, message, fix)
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
def to_h = super.compact
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
FIXTURES = %w[baseURL call context env evidence page pageAs poll request requireEnv result scratch].freeze
|
|
18
|
+
|
|
19
|
+
class Check
|
|
20
|
+
attr_reader :findings, :usable
|
|
21
|
+
|
|
22
|
+
def initialize(steps, scenarios)
|
|
23
|
+
@steps = steps
|
|
24
|
+
@scenarios = scenarios
|
|
25
|
+
@findings = []
|
|
26
|
+
@usable = steps.reject { definition_broken?(_1) }
|
|
27
|
+
@usable.each { check_contract(_1) }
|
|
28
|
+
check_parts
|
|
29
|
+
check_shapes
|
|
30
|
+
check_lines
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def passed? = findings.none? { _1.severity == "error" }
|
|
34
|
+
|
|
35
|
+
private
|
|
36
|
+
|
|
37
|
+
# A step whose definition cannot stand (an unknown claim compare) is left
|
|
38
|
+
# out of the vocabulary, as the TypeScript engine leaves out a step file
|
|
39
|
+
# that fails to import, so it can never match a line.
|
|
40
|
+
def definition_broken?(step)
|
|
41
|
+
problems = Claims.unknown_compares(step.claims || []).map { "claims compare must be one of #{Claims::COMPARES.join(", ")}; got #{_1.inspect}." }
|
|
42
|
+
problems += Definition.new(step).problems
|
|
43
|
+
problems.each do |problem|
|
|
44
|
+
findings << Finding.about_step("error", "step-file-import-failed", step, problem,
|
|
45
|
+
"Correct the error that the message names in the file, then run nuka check again.")
|
|
46
|
+
end
|
|
47
|
+
problems.any?
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def check_contract(step)
|
|
51
|
+
keys = step.args.fields.map { _1.name.to_s }
|
|
52
|
+
(step.pattern ? Pattern.new(step.pattern).keys - keys : []).each do |key|
|
|
53
|
+
findings << Finding.about_step("error", "unknown-capture-key", step,
|
|
54
|
+
%(Step "#{step.step_name}" pattern "#{step.pattern}" binds key "#{key}", which is not a key of its args schema),
|
|
55
|
+
"Rename the capture to a key of the args schema, or add the key to the schema.")
|
|
56
|
+
end
|
|
57
|
+
undescribed = [step.args, step.returns].compact.flat_map(&:fields).reject(&:description)
|
|
58
|
+
if undescribed.any?
|
|
59
|
+
findings << Finding.about_step("note", "schema-field-undescribed", step,
|
|
60
|
+
"Step \"#{step.step_name}\" has schema fields with no description: #{undescribed.map(&:name).join(", ")}",
|
|
61
|
+
"Give each field a description, so nuka steps and an agent can say what it means.")
|
|
62
|
+
end
|
|
63
|
+
known = FIXTURES + Nukadoko.project_fixtures.keys.map(&:to_s)
|
|
64
|
+
(step.fixtures.map(&:to_s) - known).each do |name|
|
|
65
|
+
findings << Finding.about_step("error", "fixture-structural-violation", step,
|
|
66
|
+
%(#{step.body}() names unknown fixture "#{name}": known fixtures are #{known.uniq.sort.join(", ")}),
|
|
67
|
+
"Name a known fixture as a keyword argument.")
|
|
68
|
+
end
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
# A cycle never closes into a fixture bag, and a read-only step cannot
|
|
72
|
+
# reach a part that writes: both can only end one way, so both are errors.
|
|
73
|
+
def check_parts
|
|
74
|
+
@usable.each do |step|
|
|
75
|
+
step.parts.reject { _1.is_a?(Class) && _1 < Step && @steps.include?(_1) }.each do |part|
|
|
76
|
+
findings << Finding.about_step("error", "part-structural-violation", step,
|
|
77
|
+
"Step \"#{step.step_name}\" declares #{part.inspect} as a part, which is not a registered step",
|
|
78
|
+
"Make each parts entry a step that a step file registers.")
|
|
79
|
+
end
|
|
80
|
+
step.parts.select { step.mutates == false && _1.respond_to?(:mutates) && _1.mutates == true }.each do |part|
|
|
81
|
+
findings << Finding.about_step("error", "part-mutates-contradiction", step,
|
|
82
|
+
"Step \"#{step.step_name}\" declares mutates: false, but its part \"#{part.step_name}\" declares mutates: true",
|
|
83
|
+
"Declare mutates: true on the step, or remove the part that declares mutates: true.")
|
|
84
|
+
end
|
|
85
|
+
end
|
|
86
|
+
part_cycles.each do |cycle|
|
|
87
|
+
findings << Finding.about_step("error", "part-cycle", cycle.first,
|
|
88
|
+
"part cycle: #{cycle.map(&:step_name).join(" -> ")}",
|
|
89
|
+
"Remove one part from the chain, so that no step reaches itself.")
|
|
90
|
+
end
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
# Each cycle once, named from the step whose walk first closed it.
|
|
94
|
+
def part_cycles
|
|
95
|
+
done = []
|
|
96
|
+
cycles = []
|
|
97
|
+
visit = lambda do |step, stack|
|
|
98
|
+
stack += [step]
|
|
99
|
+
step.parts.each do |part|
|
|
100
|
+
next unless @steps.include?(part)
|
|
101
|
+
|
|
102
|
+
if stack.include?(part) then cycles << stack.drop(stack.index(part)) + [part]
|
|
103
|
+
elsif !done.include?(part) then visit.(part, stack)
|
|
104
|
+
end
|
|
105
|
+
end
|
|
106
|
+
done << step
|
|
107
|
+
end
|
|
108
|
+
@usable.each { visit.(_1, []) unless done.include?(_1) }
|
|
109
|
+
cycles
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
# A scenario with steps of its own and no Then can never be judged on
|
|
113
|
+
# an outcome. The Background's steps count toward the Then.
|
|
114
|
+
def check_shapes
|
|
115
|
+
@scenarios.uniq { [_1.uri, _1.keyword_line] }.each do |scenario|
|
|
116
|
+
next if scenario.steps.all?(&:background) || scenario.steps.any? { _1.kind == "Outcome" }
|
|
117
|
+
|
|
118
|
+
findings << Finding.new("warning", "scenario-without-then", scenario.uri, scenario.keyword_line, nil,
|
|
119
|
+
%(Scenario "#{scenario.name}" has no Then step, so nothing about its outcome is judged),
|
|
120
|
+
"Add a Then step that states the outcome.")
|
|
121
|
+
end
|
|
122
|
+
end
|
|
123
|
+
|
|
124
|
+
def check_lines
|
|
125
|
+
undefined = []
|
|
126
|
+
@scenarios.each do |scenario|
|
|
127
|
+
scenario.steps.each do |line|
|
|
128
|
+
matches = Engine.matches(@usable, line.text)
|
|
129
|
+
case matches.size
|
|
130
|
+
when 0 then undefined << [scenario, line]
|
|
131
|
+
when 1 then check_fill(scenario, line, *matches.first)
|
|
132
|
+
else
|
|
133
|
+
findings << Finding.new("error", "ambiguous-step", scenario.uri, line.line, nil,
|
|
134
|
+
"#{matches.size} steps match \"#{line.text}\"", "Make the patterns distinct.")
|
|
135
|
+
end
|
|
136
|
+
end
|
|
137
|
+
end
|
|
138
|
+
suppress_undefined(undefined)
|
|
139
|
+
end
|
|
140
|
+
|
|
141
|
+
# While a step definition is broken, a line it would have matched looks
|
|
142
|
+
# undefined. Saying so would send the reader after the wrong fix, so the
|
|
143
|
+
# judgment waits for the broken definition, with one warning.
|
|
144
|
+
def suppress_undefined(undefined)
|
|
145
|
+
broken = @steps - @usable
|
|
146
|
+
if broken.any? && undefined.any?
|
|
147
|
+
findings << Finding.about_step("warning", "undefined-step-check-suppressed", broken.first,
|
|
148
|
+
"Suppressed #{undefined.size} undefined-step issues because #{broken.size} step file could not be read",
|
|
149
|
+
"Correct the step-file-import-failed error, then run nuka check again.")
|
|
150
|
+
return
|
|
151
|
+
end
|
|
152
|
+
undefined.each do |scenario, line|
|
|
153
|
+
findings << Finding.new("error", "undefined-step", scenario.uri, line.line, nil,
|
|
154
|
+
%(No step matches "#{line.text}"), "Add a step whose pattern matches the line.")
|
|
155
|
+
end
|
|
156
|
+
end
|
|
157
|
+
|
|
158
|
+
def check_fill(scenario, line, step, captures)
|
|
159
|
+
missing = step.args.fields.map { _1.name.to_s } - captures.keys - step.from.keys
|
|
160
|
+
missing = missing.drop(1) if (line.table || line.doc_string) && missing.any?
|
|
161
|
+
missing.each do |key|
|
|
162
|
+
findings << Finding.new("error", "unfillable-required-key", scenario.uri, line.line, step.step_name,
|
|
163
|
+
%(Step "#{step.step_name}"'s args key "#{key}" is required, but nothing on this line can fill it),
|
|
164
|
+
"Capture the key in the pattern, attach a table, or make the key optional.")
|
|
165
|
+
end
|
|
166
|
+
end
|
|
167
|
+
end
|
|
168
|
+
|
|
169
|
+
def self.matches(steps, text)
|
|
170
|
+
steps.filter_map { |step| step.pattern && (captures = Pattern.new(step.pattern).match(text)) && [step, captures] }
|
|
171
|
+
end
|
|
172
|
+
end
|
|
173
|
+
end
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# A Then step declares what it compares; the engine judges. A claim names the
|
|
4
|
+
# actual value (a key the step observed or was given, and an optional `at`
|
|
5
|
+
# path into it), a compare, and where the expected value comes from: an args
|
|
6
|
+
# key, `{ literal: v }`, or `{ from: key }` for an args key an earlier step
|
|
7
|
+
# filled. The judge returns every comparison it made, so a broken claim shows
|
|
8
|
+
# what was read.
|
|
9
|
+
module Nukadoko
|
|
10
|
+
module Engine
|
|
11
|
+
module Claims
|
|
12
|
+
COMPARES = %w[equal not-equal contains-text starts-with-text ends-with-text has-item has-same-items has-fields
|
|
13
|
+
lacks-field fits-text fits-aria-snapshot is-a matches greater-than at-least less-than at-most judged-as].freeze
|
|
14
|
+
|
|
15
|
+
def self.unknown_compares(node)
|
|
16
|
+
case node
|
|
17
|
+
in Array then node.flat_map { unknown_compares(_1) }
|
|
18
|
+
in Hash
|
|
19
|
+
own = node.key?(:compare) && !COMPARES.include?(node[:compare]) ? [node[:compare]] : []
|
|
20
|
+
own + node.values.flat_map { unknown_compares(_1) }
|
|
21
|
+
else []
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def self.judge(claims, observed, args)
|
|
26
|
+
claims.map { judge_one(_1, observed, args) }
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
def self.judge_one(claim, observed, args)
|
|
30
|
+
actual = dig(view(claim[:view], read(observed, claim[:actual])), claim[:at])
|
|
31
|
+
return judge_each(claim[:each], actual, args) if claim[:each]
|
|
32
|
+
|
|
33
|
+
expected = expected_value(claim, args)
|
|
34
|
+
{ actual: actual, compare: claim[:compare], expected: expected, holds: holds?(claim[:compare], actual, expected) }
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
def self.judge_each(each, items, args)
|
|
38
|
+
chosen = Array(items).select { |item| each[:where].all? { judge_one(_1, item, args)[:holds] } }
|
|
39
|
+
results = chosen.flat_map { |item| each[:claims].map { judge_one(_1, item, args) } }
|
|
40
|
+
{ each: results, chosen: chosen.size, holds: !chosen.empty? && results.all? { _1[:holds] } }
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
def self.expected_value(claim, args)
|
|
44
|
+
value =
|
|
45
|
+
case claim[:expected]
|
|
46
|
+
in String => key then read(args, key)
|
|
47
|
+
in { literal: literal } then literal
|
|
48
|
+
in { from: String => key } then read(args, key)
|
|
49
|
+
end
|
|
50
|
+
claim[:expected_view] == "json" ? JSON.parse(value) : value
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
VIEWS = %w[text lines comma-list json].freeze
|
|
54
|
+
|
|
55
|
+
# A view reads a string as something else before `at` walks into it.
|
|
56
|
+
def self.view(name, value)
|
|
57
|
+
return value unless name
|
|
58
|
+
raise ArgumentError, "view #{name} reads a string, and the value is #{value.inspect}" unless value.is_a?(String)
|
|
59
|
+
|
|
60
|
+
case name
|
|
61
|
+
when "text" then value.rstrip
|
|
62
|
+
when "lines" then value.rstrip.empty? ? [] : value.rstrip.split(/\r?\n/)
|
|
63
|
+
when "comma-list" then value.strip.empty? ? [] : value.strip.split(",").map(&:strip)
|
|
64
|
+
when "json" then JSON.parse(value)
|
|
65
|
+
else raise ArgumentError, "unknown view #{name}"
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def self.read(hash, key)
|
|
70
|
+
hash = hash.transform_keys(&:to_s) if hash.is_a?(Hash)
|
|
71
|
+
hash.fetch(key.to_s)
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
def self.dig(value, path)
|
|
75
|
+
return value unless path
|
|
76
|
+
|
|
77
|
+
path.to_s.split(".").reduce(value) { |v, part| v.is_a?(Array) ? v[Integer(part)] : read(v, part) }
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
def self.same?(a, b) = normal(a) == normal(b)
|
|
81
|
+
|
|
82
|
+
def self.normal(value)
|
|
83
|
+
case value
|
|
84
|
+
when Hash then value.to_h { |k, v| [k.to_s, normal(v)] }
|
|
85
|
+
when Array then value.map { normal(_1) }
|
|
86
|
+
when Symbol then value.to_s
|
|
87
|
+
else value
|
|
88
|
+
end
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def self.holds?(compare, actual, expected)
|
|
92
|
+
case compare
|
|
93
|
+
when "equal" then same?(actual, expected)
|
|
94
|
+
when "not-equal" then !same?(actual, expected)
|
|
95
|
+
when "contains-text" then actual.to_s.include?(expected.to_s)
|
|
96
|
+
when "starts-with-text" then actual.to_s.start_with?(expected.to_s)
|
|
97
|
+
when "ends-with-text" then actual.to_s.end_with?(expected.to_s)
|
|
98
|
+
when "has-item" then Array(actual).any? { same?(_1, expected) }
|
|
99
|
+
when "has-same-items" then Array(actual).map { normal(_1) }.tally == Array(expected).map { normal(_1) }.tally
|
|
100
|
+
when "greater-than" then actual.is_a?(Numeric) && actual > expected
|
|
101
|
+
when "at-least" then actual.is_a?(Numeric) && actual >= expected
|
|
102
|
+
when "less-than" then actual.is_a?(Numeric) && actual < expected
|
|
103
|
+
when "at-most" then actual.is_a?(Numeric) && actual <= expected
|
|
104
|
+
when "matches" then Regexp.new(expected.to_s).match?(actual.to_s)
|
|
105
|
+
else raise NotImplementedError, "the Ruby engine cannot judge #{compare} yet"
|
|
106
|
+
end
|
|
107
|
+
end
|
|
108
|
+
end
|
|
109
|
+
end
|
|
110
|
+
end
|
data/lib/nukadoko/cli.rb
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# The `nuka` command. With --json it prints one envelope in the shape the
|
|
4
|
+
# TypeScript engine prints, so an agent reads both engines the same way.
|
|
5
|
+
module Nukadoko
|
|
6
|
+
class CLI
|
|
7
|
+
def initialize(argv, project: Dir.pwd, out: $stdout)
|
|
8
|
+
@command, *rest = argv
|
|
9
|
+
@json = rest.delete("--json")
|
|
10
|
+
@no_check = rest.delete("--no-check")
|
|
11
|
+
@rest = rest
|
|
12
|
+
@project = File.expand_path(project)
|
|
13
|
+
@out = out
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
def call
|
|
17
|
+
%w[nukadoko.config.rb nukadoko.program.rb].each do |name|
|
|
18
|
+
file = File.join(@project, name)
|
|
19
|
+
load file if File.exist?(file)
|
|
20
|
+
end
|
|
21
|
+
steps = Nukadoko.load_steps(File.join(@project, "features", "steps"))
|
|
22
|
+
scenarios = Dir.chdir(@project) { Engine.load_features("features") }
|
|
23
|
+
check = Engine::Check.new(steps, scenarios)
|
|
24
|
+
findings = check.findings.map { relative(_1).to_h }
|
|
25
|
+
case @command
|
|
26
|
+
when "check" then finish("check", check.passed?, findings: findings)
|
|
27
|
+
when "sig"
|
|
28
|
+
path = File.join(@project, "sig", "nukadoko", "steps.rbs")
|
|
29
|
+
FileUtils.mkdir_p(File.dirname(path))
|
|
30
|
+
File.write(path, Signature.of(steps))
|
|
31
|
+
@out.puts "wrote #{path.delete_prefix("#{@project}/")}"
|
|
32
|
+
0
|
|
33
|
+
when "steps"
|
|
34
|
+
entries = check.usable.sort_by(&:file)
|
|
35
|
+
return finish("steps", true, findings: findings, steps: entries.map { Vocabulary.entry(_1, @project) }) if @json
|
|
36
|
+
|
|
37
|
+
@out.puts entries.map { Vocabulary.text(_1) }.join("\n\n")
|
|
38
|
+
0
|
|
39
|
+
when "describe"
|
|
40
|
+
step = check.usable.find { _1.step_name == @rest.first }
|
|
41
|
+
@json = true
|
|
42
|
+
unless step
|
|
43
|
+
refusal = { severity: "refused", code: "unknown-step", message: "Unknown step: #{@rest.first}", fix: "Name a step that nuka steps lists.", do: "nuka steps" }
|
|
44
|
+
return finish("describe", false, findings: findings + [refusal], verdict: "refused")
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
finish("describe", true, findings: findings, **Vocabulary.describe(step, @project))
|
|
48
|
+
when "run"
|
|
49
|
+
return finish("run", false, findings: findings) unless @no_check || check.passed?
|
|
50
|
+
|
|
51
|
+
result = Engine::Run.new(@project, check.usable, scenarios, Nukadoko.program_options, out: @json ? $stderr : @out).call
|
|
52
|
+
passed = result.delete(:passed)
|
|
53
|
+
finish("run", passed, findings: findings, **result)
|
|
54
|
+
else
|
|
55
|
+
warn "usage: nuka check|run|steps|describe <step>|sig [--json] [--no-check]"
|
|
56
|
+
2
|
|
57
|
+
end
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
private
|
|
61
|
+
|
|
62
|
+
def relative(finding)
|
|
63
|
+
finding.with(file: finding.file&.delete_prefix("#{@project}/"))
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
def finish(command, passed, findings:, verdict: passed ? "passed" : "failed", **rest)
|
|
67
|
+
if @json
|
|
68
|
+
@out.puts JSON.pretty_generate({ nukadoko: VERSION, engine: "ruby", command: command, verdict: verdict, findings: findings, **rest })
|
|
69
|
+
else
|
|
70
|
+
findings.each { @out.puts [_1[:severity], _1[:code], [_1[:file], _1[:line]].compact.join(":"), _1[:message]].join("\t") }
|
|
71
|
+
@out.puts "ok: no issues found" if command == "check" && findings.empty?
|
|
72
|
+
end
|
|
73
|
+
passed ? 0 : 1
|
|
74
|
+
end
|
|
75
|
+
end
|
|
76
|
+
end
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "open3"
|
|
4
|
+
require "tmpdir"
|
|
5
|
+
require "fileutils"
|
|
6
|
+
|
|
7
|
+
# What a command-line scenario needs: scratch directories that live as long
|
|
8
|
+
# as one scenario, and a way to run the program under test that keeps every
|
|
9
|
+
# run for the step record. Both are engine-built, so a step never removes a
|
|
10
|
+
# directory it made or forgets to note a process it started.
|
|
11
|
+
module Nukadoko
|
|
12
|
+
# Runs a command and gives what it left: the exit code (nil when a signal
|
|
13
|
+
# ended it), stdout, and stderr. Inside a step, the run is noted in the
|
|
14
|
+
# step record's processes.
|
|
15
|
+
def self.run_command(command, args = [], cwd: Dir.pwd, env: {}, stdin: nil)
|
|
16
|
+
started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
17
|
+
stdout, stderr, status = Open3.capture3(env.transform_keys(&:to_s), command, *args, chdir: cwd, stdin_data: stdin.to_s)
|
|
18
|
+
result = { exitCode: status.exitstatus, stdout: stdout, stderr: stderr }
|
|
19
|
+
Engine::Processes.current&.push(
|
|
20
|
+
command: command, args: args, cwd: File.expand_path(cwd), exit_code: status.exitstatus,
|
|
21
|
+
signal: status.termsig && Signal.signame(status.termsig),
|
|
22
|
+
duration_ms: ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) * 1000).round
|
|
23
|
+
)
|
|
24
|
+
result
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
module Engine
|
|
28
|
+
# The processes the running step started, kept per thread so a step
|
|
29
|
+
# that calls run_command from a helper is still measured.
|
|
30
|
+
module Processes
|
|
31
|
+
def self.current = Thread.current[:nukadoko_processes]
|
|
32
|
+
|
|
33
|
+
# Fills `into` even when the step raises, so a failed step still names
|
|
34
|
+
# the processes it ran.
|
|
35
|
+
def self.collect(into)
|
|
36
|
+
Thread.current[:nukadoko_processes] = into
|
|
37
|
+
yield
|
|
38
|
+
ensure
|
|
39
|
+
Thread.current[:nukadoko_processes] = nil
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
# The `scratch` fixture: `dir` makes a fresh directory, and the engine
|
|
44
|
+
# removes every one of them when the scenario ends, passed or failed.
|
|
45
|
+
class Scratch
|
|
46
|
+
def initialize
|
|
47
|
+
@dirs = []
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def dir(prefix = "nukadoko-scratch")
|
|
51
|
+
raise ArgumentError, "a scratch prefix cannot hold a path separator: #{prefix}" if prefix.match?(%r{[/\\]})
|
|
52
|
+
|
|
53
|
+
Dir.mktmpdir(prefix).tap { @dirs << _1 }
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def dirs = @dirs.dup
|
|
57
|
+
|
|
58
|
+
def remove_all
|
|
59
|
+
@dirs.each { FileUtils.rm_rf(_1) }
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
end
|
|
63
|
+
end
|