plai_testr 0.2.0 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +10 -1
- data/README.md +20 -194
- data/docs/AGENTS.md +6 -5
- data/docs/getting-started.md +137 -0
- data/docs/ingesting-a-game-manually.md +112 -0
- data/docs/ingesting-a-game-with-an-agent.md +138 -0
- data/docs/running-playtests.md +193 -0
- data/docs/technical/README.md +15 -0
- data/docs/technical/debrief.md +55 -0
- data/docs/technical/personas-and-bots.md +67 -0
- data/docs/technical/reports.md +57 -0
- data/docs/technical/rules-and-components.md +61 -0
- data/docs/technical/runner.md +158 -0
- data/docs/technical/specs.md +62 -0
- data/docs/technical/state.md +60 -0
- data/lib/plai_testr/cli.rb +1 -1
- data/lib/plai_testr/examples.rb +37 -0
- data/lib/plai_testr/generator.rb +3 -2
- data/lib/plai_testr/reporter/botrun_report.rb +1 -1
- data/lib/plai_testr/reporter/duration.rb +12 -0
- data/lib/plai_testr/reporter/report_builder.rb +1 -0
- data/lib/plai_testr/session/runner.rb +3 -1
- data/lib/plai_testr/state/game_state.rb +1 -1
- data/lib/plai_testr/version.rb +1 -1
- data/lib/plai_testr.rb +2 -0
- data/templates/app/AGENTS.md.erb +1 -1
- data/templates/app/README.md.erb +3 -3
- data/templates/examples/gin_rummy/data/components/cards.csv +53 -0
- data/templates/examples/gin_rummy/data/rules.json +1 -64
- metadata +18 -3
- /data/docs/{frame_queue.md → technical/frame-queue.md} +0 -0
- /data/docs/{referee.md → technical/referee.md} +0 -0
|
@@ -0,0 +1,158 @@
|
|
|
1
|
+
# The runner
|
|
2
|
+
|
|
3
|
+
A game's runner subclasses `PlaiTestr::Session::Runner`. The gem owns the decision loop; the runner supplies the rules through *hooks*, public methods the loop calls.
|
|
4
|
+
|
|
5
|
+
## The decision loop
|
|
6
|
+
|
|
7
|
+
Each decision:
|
|
8
|
+
|
|
9
|
+
1. `next_player` says who acts.
|
|
10
|
+
2. `@rng` is reseeded for this decision (see [Randomness](#randomness)).
|
|
11
|
+
3. `legal_moves(player)` builds the menu, and `pick_range(player)` says whether it's a choose-one or choose-several decision. The gem checks the menu (see [Moves](#moves-and-actions)).
|
|
12
|
+
4. Someone decides:
|
|
13
|
+
- A menu with one option is applied without asking anyone.
|
|
14
|
+
- Otherwise the model gets the persona, rules text, view, recent events and the numbered menu, and replies with a choice, its reasoning and its plan. In `--mode hybrid`, an *obvious* bot move is played without asking the model. In `--mode bot` and botruns, the bot decides.
|
|
15
|
+
5. `apply_action_effects(player, action)` applies the chosen move.
|
|
16
|
+
6. The game is saved, and the loop repeats until `game_over?`.
|
|
17
|
+
|
|
18
|
+
Then each model player is debriefed (see [Debrief](debrief.md)) and the report is written.
|
|
19
|
+
|
|
20
|
+
Registration ties the runner to the game's other classes, in `lib/<name>.rb`:
|
|
21
|
+
|
|
22
|
+
```ruby
|
|
23
|
+
PlaiTestr.register('hearts',
|
|
24
|
+
runner_class: Hearts::Session::Runner,
|
|
25
|
+
game_state_class: Hearts::State::GameState,
|
|
26
|
+
botrun_report_class: Hearts::Reporter::BotrunReport, # optional, see Reports
|
|
27
|
+
players: ['Player 1', 'Player 2', 'Player 3', 'Player 4'],
|
|
28
|
+
recent_events: 12, # log lines per prompt (default 8)
|
|
29
|
+
max_decisions: 5_000, # this game's decision limit
|
|
30
|
+
model_settings: { context_size: 12_288 }) # this game's model settings
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
`players` is the default seating when `rules.json` has no `player_count`. `comparison_class:` swaps in a subclass of `Reporter::ModelComparison` for `compare`.
|
|
34
|
+
|
|
35
|
+
## Hooks
|
|
36
|
+
|
|
37
|
+
| Hook | Required | Purpose |
|
|
38
|
+
|---|---|---|
|
|
39
|
+
| `setup` | yes | Build the starting state. Not called on resume. |
|
|
40
|
+
| `legal_moves(player)` | yes | The menu: an array of `option(...)` entries |
|
|
41
|
+
| `apply_action_effects(player, action)` | yes | Apply the chosen move's action to `@game_state` |
|
|
42
|
+
| `view_text_for(player)` | expected | The player's view of the game, as text |
|
|
43
|
+
| `debrief_facts(player)` | expected | Engine-counted facts for the debrief, as a Hash (default: `botrun_stats`; see [Debrief](debrief.md)) |
|
|
44
|
+
| `next_player` | | Who acts next (default: alternate) |
|
|
45
|
+
| `pick_range(player)` | | `[min, max]` to pick several options at once |
|
|
46
|
+
| `game_over?` | | Default: `status['game_over']` |
|
|
47
|
+
| `turn_phase(player)` | | Short step label for the console and report |
|
|
48
|
+
| `rules_text_for(player)` | | Rules text (default: the game's `data/rules.md`; call `super` to extend it) |
|
|
49
|
+
| `prompt_note_for(player)` | | A one-off note for this prompt |
|
|
50
|
+
| `recent_events_for(player)` | | Recent events from the event log (default: the last 8 this player may see; `recent_events:` in `register`) |
|
|
51
|
+
| `log_turn`, `log_turn_name` | | The number stamped on log entries and its name (default: the decision number) |
|
|
52
|
+
| `persona_for(player)` | | Strategy text for the system prompt (default: built from `personas.yml`) |
|
|
53
|
+
| `display_name(player)` | | Name in the console and reports (default: the persona's key, title-cased, else the player id) |
|
|
54
|
+
| `bot_choice(player, moves, pick: nil)` | | Bot rules (see [Personas and bots](personas-and-bots.md)) |
|
|
55
|
+
| `think_about?(player)` | | With `--think`: whether this turn gets thinking (default: every turn) |
|
|
56
|
+
| `debrief_questions(player)` | | End-of-game feedback questions |
|
|
57
|
+
| `debrief_view_for(player)` | | The final position the debrief reads (default: the result, final scores, `view_text_for` and the player's visible log) |
|
|
58
|
+
| `debrief_summary(player)` | | Built by the gem, not overridden: the summary that opens each debrief prompt |
|
|
59
|
+
| `report_sections`, `report_debrief?` | | Extra report sections; false to render the debrief yourself (see [Reports](reports.md)) |
|
|
60
|
+
| `botrun_stats` | | Game-specific numbers for one botrun game (see [Reports](reports.md)) |
|
|
61
|
+
|
|
62
|
+
Hooks are public, so specs can call them directly; the helpers below are private. Put your own helpers below `private` too. Avoid public helper names that could collide with future hooks.
|
|
63
|
+
|
|
64
|
+
### Helpers
|
|
65
|
+
|
|
66
|
+
Available inside hooks:
|
|
67
|
+
|
|
68
|
+
| Helper | |
|
|
69
|
+
|---|---|
|
|
70
|
+
| `option(label, public:, log:, **action)` | Build a menu entry (see below) |
|
|
71
|
+
| `menu(player)` | The menu as the runner sees it, after checking |
|
|
72
|
+
| `data` | `@game_state.data`, the game's own state |
|
|
73
|
+
| `log(text, to: nil)` | Add an event to the event log; `to:` makes it private to one player |
|
|
74
|
+
| `event_log` | The log itself |
|
|
75
|
+
| `players`, `pstate(player)` | Player names in seat order; one player's `PlayerState` |
|
|
76
|
+
| `whose(player, viewer)` | "You" when the player is the viewer, else the player's name, for view text |
|
|
77
|
+
| `tally(player, key, amount = 1)` | Count something per player in `data['tally']` (for facts and reports) |
|
|
78
|
+
| `sole_leader(scores = @game_state.scores)` | The single highest scorer, or `nil` on a tie |
|
|
79
|
+
| `persona(player)`, `persona_name(player)`, `bot_strategy_for(player)` | The player's persona key, its display name, its bot parameters |
|
|
80
|
+
| `@rng`, `@rules`, `@game_state`, `out` | Random generator, parsed `rules.json`, state, console output |
|
|
81
|
+
|
|
82
|
+
## Moves and actions
|
|
83
|
+
|
|
84
|
+
```ruby
|
|
85
|
+
def legal_moves(player)
|
|
86
|
+
hand(player).map { |card| option("Play #{card['name']}", public: "played #{card['name']}", play: { value: card['uid'] }) }
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
def apply_action_effects(player, action)
|
|
90
|
+
case action
|
|
91
|
+
in { play: { value: uid } } then play(player, uid)
|
|
92
|
+
in { pass: { value: uids } } then pass(player, uids)
|
|
93
|
+
end
|
|
94
|
+
end
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
- `label` (the first argument) is what the acting player sees; `public:` is what the others see; `log:` is a shorter console line.
|
|
98
|
+
- The action has exactly one name (`play:` above). `value:` is its main parameter; others sit beside it (`play: { value: uid, target: 2 }`).
|
|
99
|
+
- Moves and actions have symbol keys, so `apply_action_effects` and the bot can pattern-match them. A hand-written move with string keys is converted.
|
|
100
|
+
- On a pick turn (`pick_range` returns `[min, max]`) the chosen values arrive together: `{ pass: { value: [42, 17, 9] } }`.
|
|
101
|
+
- Every menu is checked before it's used. A misspelled key or a malformed action raises `PlaiTestr::MoveError` naming the hook, player, decision and option; repeated labels get a warning.
|
|
102
|
+
|
|
103
|
+
### Writing good menus
|
|
104
|
+
|
|
105
|
+
Small local models play well only when the engine does the bookkeeping:
|
|
106
|
+
|
|
107
|
+
- **The engine lists the legal moves; the model picks a number.** Never ask the model to name a card or check legality.
|
|
108
|
+
- **Put engine-computed facts on options, never advice.** "Discard 9♥ (deadwood after: 31)" is a fact; "Discard 9♥ (best)" decides for the model.
|
|
109
|
+
- **Generate views from state**, never hand-written summaries, which go stale or leak hidden information.
|
|
110
|
+
- **Keep menus short.** Merge equivalent options (identical cards, interchangeable targets); split a huge decision into steps.
|
|
111
|
+
- **Hide safety caps.** Turn or hand limits that stop runaway games must not appear in views or rules text, or models play for the cap.
|
|
112
|
+
- **Scope long rules.** Override `rules_text_for` to send only the sections the current step needs.
|
|
113
|
+
|
|
114
|
+
## Steps
|
|
115
|
+
|
|
116
|
+
Most games model a turn as a small state machine in `data['step']` (`draw` → `discard` → next player). `legal_moves` and `apply_action_effects` switch on the step, and `turn_phase` usually shows it. For decisions inside decisions (an attack the defender may answer, a triggered effect), use [FrameQueue](frame-queue.md).
|
|
117
|
+
|
|
118
|
+
## Ending the game
|
|
119
|
+
|
|
120
|
+
Set the status when the game ends:
|
|
121
|
+
|
|
122
|
+
```ruby
|
|
123
|
+
@game_state.apply_changes('status' => { 'game_over' => true, 'winner' => leader, 'end_reason' => 'points' })
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
`winner` is `nil` for a tie. `end_reason` is a short key, shown in reports and counted by botruns.
|
|
127
|
+
|
|
128
|
+
## Randomness
|
|
129
|
+
|
|
130
|
+
Use `@rng` for all randomness (`deck.shuffle(random: @rng)`, `@rng.rand(6) + 1`). It is reseeded for each decision from the game's seed and the decision number, so the same seed and the same choices always give the same game, interrupted or not. Any other source of randomness breaks replays and resume.
|
|
131
|
+
|
|
132
|
+
## Limits and fallbacks
|
|
133
|
+
|
|
134
|
+
- Every game stops at a decision limit, ending with `end_reason: 'decision_limit_reached'` and no winner, so a buggy game can't run forever. Set `max_decisions:` in `register`; otherwise the app's `max_decisions` applies.
|
|
135
|
+
- When the model never gives a valid answer after retries, the bot's choice is played instead (or a random legal move, from a separate generator, when the bot has none), and the turn is flagged in the report.
|
|
136
|
+
- A bot rule that raises in a model game is skipped with a warning; in a botrun it is recorded as a crash for that seed.
|
|
137
|
+
|
|
138
|
+
## The event log
|
|
139
|
+
|
|
140
|
+
`log(text)` records a public event; `log(text, to: player)` a private one ("You drew the 2♠"). Each entry is stamped with `log_turn`: the decision number by default. A game with rounds returns its round number, with `log_turn_name` its word for it ("round", "hand"). `recent_events_for` shows each player the last events they may see, and the debrief view shows the player's whole visible log.
|
|
141
|
+
|
|
142
|
+
## Checking a prompt
|
|
143
|
+
|
|
144
|
+
Read what the model sees before running a model game:
|
|
145
|
+
|
|
146
|
+
```ruby
|
|
147
|
+
# bundle exec ruby -e '...'
|
|
148
|
+
require 'plai_testr'
|
|
149
|
+
game = PlaiTestr.load('hearts')
|
|
150
|
+
runner = game.runner_class.new(game: game, players: game.players, seed: 1, mode: 'bot', persist: false, out: StringIO.new)
|
|
151
|
+
runner.setup
|
|
152
|
+
5.times { runner.play_turn }
|
|
153
|
+
player = runner.next_player
|
|
154
|
+
puts runner.persona_for(player), runner.view_text_for(player), runner.menu(player).map { |m| m[:label] }
|
|
155
|
+
|
|
156
|
+
runner.play_turn until runner.game_over?
|
|
157
|
+
puts runner.debrief_summary(player), runner.debrief_view_for(player)
|
|
158
|
+
```
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# Specs
|
|
2
|
+
|
|
3
|
+
Each game's specs are in `games/<name>/spec/` and run with [RSpec](https://rspec.info):
|
|
4
|
+
|
|
5
|
+
```sh
|
|
6
|
+
bundle exec rspec # every game
|
|
7
|
+
bundle exec rspec games/hearts/spec # one game
|
|
8
|
+
```
|
|
9
|
+
|
|
10
|
+
They need no model and run in seconds, so run them after every change.
|
|
11
|
+
|
|
12
|
+
## The standard checks
|
|
13
|
+
|
|
14
|
+
Every game includes the gem's shared checks (from `plai_testr/rspec`, which the app's `spec_helper` requires):
|
|
15
|
+
|
|
16
|
+
```ruby
|
|
17
|
+
RSpec.describe Hearts::Session::Runner do
|
|
18
|
+
it_behaves_like 'a PlAI Testr game', 'hearts'
|
|
19
|
+
end
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
At every player count in `rules.json`, over a few seeds, they check that:
|
|
23
|
+
|
|
24
|
+
- bot games reach an ending
|
|
25
|
+
- every turn prompt builds, and every debrief prompt at the end (botruns never build prompts, so this is the only check before a model game)
|
|
26
|
+
- state survives a JSON round trip
|
|
27
|
+
- resuming from a mid-game checkpoint gives the same result
|
|
28
|
+
- no player's prompt shows another player's hand
|
|
29
|
+
|
|
30
|
+
Options: `seeds: [1, 2, 3]` sets which games are played; `pending: 'reason'` holds the checks while a new game is being built (the generated spec starts with it).
|
|
31
|
+
|
|
32
|
+
The checks come from the installed gem, so upgrading the gem adds checks without editing the game. Don't copy them into a game.
|
|
33
|
+
|
|
34
|
+
## Rules specs
|
|
35
|
+
|
|
36
|
+
Rules specs drive the runner directly. Build it without saving anything, set up a situation, and check the menu or the effects:
|
|
37
|
+
|
|
38
|
+
```ruby
|
|
39
|
+
RSpec.describe Hearts::Session::Runner do
|
|
40
|
+
let(:game) { PlaiTestr.load('hearts') }
|
|
41
|
+
let(:runner) { described_class.new(game: game, players: game.players, seed: 1, mode: 'bot', persist: false, out: StringIO.new) }
|
|
42
|
+
let(:state) { runner.game_state }
|
|
43
|
+
let(:deck) { PlaiTestr::Deck.build(game.rules) }
|
|
44
|
+
|
|
45
|
+
def cards(*names) = names.map { |name| deck.find { |c| c['name'] == name } }
|
|
46
|
+
|
|
47
|
+
def labels(player) = runner.legal_moves(player).map { |m| m[:label] }
|
|
48
|
+
|
|
49
|
+
before { runner.setup }
|
|
50
|
+
|
|
51
|
+
it 'follows the suit led when it can' do
|
|
52
|
+
state.data.merge!('step' => 'play', 'tricks' => 1, 'active' => 'Player 2', 'trick' => [{ 'player' => 'Player 1', 'card' => cards('9♣').first }])
|
|
53
|
+
state.player_state('Player 2').hand.replace(cards('5♥', '7♣', 'K♣'))
|
|
54
|
+
|
|
55
|
+
expect(labels('Player 2')).to eq(['Play 7♣ (stays under the 9♣)', 'Play K♣ (wins the trick so far)'])
|
|
56
|
+
end
|
|
57
|
+
end
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Hooks are public, so specs call them directly (the runner's private helpers such as `pstate` aren't; use `runner.game_state`). `runner.play_turn` and `runner.play_out` play one decision or the rest of the game. Write rules specs before any model plays: rules bugs are cheapest here.
|
|
61
|
+
|
|
62
|
+
Add specs for what's specific to the game too: card counts, report sections, bot rules (`spec/bot_spec.rb`). If `data/` is copied from another repository, add a spec that fails when the copy is behind.
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# State
|
|
2
|
+
|
|
3
|
+
Everything a game knows lives in `@game_state` ([`game_state.rb`](../../lib/plai_testr/state/game_state.rb)), so the gem can save it after every decision and restore it exactly.
|
|
4
|
+
|
|
5
|
+
## Game state
|
|
6
|
+
|
|
7
|
+
`@game_state` holds:
|
|
8
|
+
|
|
9
|
+
- `turn`: decisions so far
|
|
10
|
+
- `players`: player names, in seat order
|
|
11
|
+
- `status`: owned by the runner. Set `game_over`, `winner` and `end_reason` when the game ends, with `@game_state.apply_changes('status' => {...})`.
|
|
12
|
+
- `data`: the game's own state
|
|
13
|
+
|
|
14
|
+
Each `player_state(name)` ([`player_state.rb`](../../lib/plai_testr/state/player_state.rb)), or `pstate(name)` in the runner, holds `score`, `hand` and its own `data`.
|
|
15
|
+
|
|
16
|
+
### Rules for `data`
|
|
17
|
+
|
|
18
|
+
- **Plain JSON values only**: hashes with string keys, arrays, strings, numbers, booleans, `nil`. Saving is a JSON dump, so anything else (symbols, objects, sets) breaks resume.
|
|
19
|
+
- **Stable ids.** Give every card a `uid` (`Deck.build` does) and keep pile and slot indexes fixed. Never re-index things players refer to.
|
|
20
|
+
- **Counters belong in `data` too**, so reports and debrief facts survive a resume (`tally` stores them in `data['tally']`).
|
|
21
|
+
- **No instance variables for game state.** They're gone after a resume or when `debrief` rebuilds a report.
|
|
22
|
+
|
|
23
|
+
### Naming your keys
|
|
24
|
+
|
|
25
|
+
Subclass `GameState` to give `data` keys readers:
|
|
26
|
+
|
|
27
|
+
```ruby
|
|
28
|
+
module Hearts
|
|
29
|
+
module State
|
|
30
|
+
class GameState < PlaiTestr::State::GameState
|
|
31
|
+
def step = @data['step']
|
|
32
|
+
def trick = @data['trick'] ||= []
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Register the subclass with `game_state_class:`.
|
|
39
|
+
|
|
40
|
+
## Hidden information
|
|
41
|
+
|
|
42
|
+
Each prompt is built from what that player may know. Two methods decide it:
|
|
43
|
+
|
|
44
|
+
- **`GameState#data_for(observer)`** (private) returns the `data` an observer may see. The default already hides the event log entries meant for other players and the game's internal warnings. Extend it to hide decks or secret choices:
|
|
45
|
+
|
|
46
|
+
```ruby
|
|
47
|
+
def data_for(observer)
|
|
48
|
+
super.except('deck', 'passes').merge('deck_size' => @data['deck'].length)
|
|
49
|
+
end
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
- **`PlayerState#serialize_for(observer)`** shows a player their whole state and others only `name`, `score`, `hand_size` and `public_data`. Subclass `PlayerState` and override `public_data` to show opponents more (a face-up card, a pile size), and override `player_state_class` in your `GameState` to use it.
|
|
53
|
+
|
|
54
|
+
The [standard checks](specs.md) verify that no player's prompt shows another player's hand.
|
|
55
|
+
|
|
56
|
+
## Checkpoints and resume
|
|
57
|
+
|
|
58
|
+
After every decision the gem replaces `sessions/<id>/state.json` with the full state, the turn log, and a *fingerprint*: a hash of the game's `data/` and `lib/` files and the gem version. `resume` restores the state and continues from the next decision; since `@rng` is reseeded per decision, the game plays exactly as it would have without the interruption.
|
|
59
|
+
|
|
60
|
+
`resume` refuses when the fingerprint differs: changed code or data might replay differently, and a mismatched replay would make the report misleading. `debrief` warns instead, since it only rebuilds the debrief.
|
data/lib/plai_testr/cli.rb
CHANGED
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
require 'yaml'
|
|
2
|
+
|
|
3
|
+
module PlaiTestr
|
|
4
|
+
module Examples
|
|
5
|
+
DIR = File.expand_path('../../templates/examples', __dir__)
|
|
6
|
+
STAMP = '.plai_testr.yml'
|
|
7
|
+
SKIPPED = %w[sessions reports].freeze
|
|
8
|
+
|
|
9
|
+
def self.names = Dir.children(DIR).sort
|
|
10
|
+
|
|
11
|
+
def self.checksum(dir)
|
|
12
|
+
files = Dir.glob('**/*', base: dir).reject { |path| SKIPPED.include?(path.split('/').first) }.select { |path| File.file?(File.join(dir, path)) }.sort
|
|
13
|
+
digest = Digest::SHA256.new
|
|
14
|
+
files.each { |path| digest << path << "\0" << File.binread(File.join(dir, path)) << "\0" }
|
|
15
|
+
digest.hexdigest
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def self.template_checksum(name) = checksum(File.join(DIR, name))
|
|
19
|
+
|
|
20
|
+
def self.stamp(game_dir, name)
|
|
21
|
+
File.write(File.join(game_dir, STAMP), <<~YAML)
|
|
22
|
+
# Written by plai_testr when it copied this example game. A gem update replaces the game
|
|
23
|
+
# only while its files still match this checksum; once edited, the new version goes alongside.
|
|
24
|
+
#{YAML.dump('example' => name, 'version' => VERSION, 'checksum' => template_checksum(name)).delete_prefix("---\n").chomp}
|
|
25
|
+
YAML
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def self.stamp_for(game_dir) = YAML.safe_load_file(File.join(game_dir, STAMP))
|
|
29
|
+
|
|
30
|
+
def self.replaceable?(game_dir) = stamp_for(game_dir)['checksum'] == checksum(game_dir)
|
|
31
|
+
|
|
32
|
+
def self.current?(game_dir)
|
|
33
|
+
stamp = stamp_for(game_dir)
|
|
34
|
+
checksum(game_dir) == template_checksum(stamp['example'])
|
|
35
|
+
end
|
|
36
|
+
end
|
|
37
|
+
end
|
data/lib/plai_testr/generator.rb
CHANGED
|
@@ -15,8 +15,9 @@ module PlaiTestr
|
|
|
15
15
|
raise UsageError, "#{dir} already exists and is not empty" if Dir.exist?(root) && !Dir.empty?(root)
|
|
16
16
|
|
|
17
17
|
copy(File.join(TEMPLATES, 'app'), root, app_name: File.basename(root), version: VERSION.split('.').first(2).join('.'))
|
|
18
|
-
|
|
19
|
-
copy(File.join(
|
|
18
|
+
Examples.names.each do |example|
|
|
19
|
+
copy(File.join(Examples::DIR, example), File.join(root, 'games', example))
|
|
20
|
+
Examples.stamp(File.join(root, 'games', example), example)
|
|
20
21
|
end
|
|
21
22
|
Guide.link(root)
|
|
22
23
|
root
|
|
@@ -32,7 +32,7 @@ module PlaiTestr
|
|
|
32
32
|
"| **Players** | #{@played.first&.dig('names')&.length || '—'} |"
|
|
33
33
|
]
|
|
34
34
|
rows << "| **Personas** | #{@personas ? @personas.join(', ') : 'random per seed'} |" if personas?
|
|
35
|
-
rows << "| **
|
|
35
|
+
rows << "| **Wall time** | #{Duration.format(@seconds)} (#{(@seconds * 1000 / [@results.length, 1].max).round} ms per game) |"
|
|
36
36
|
rows << "| **Generated** | #{Time.now.strftime('%Y-%m-%d %H:%M')} |"
|
|
37
37
|
|
|
38
38
|
"# Botrun: #{@game}\n\n| | |\n|---|---|\n#{rows.join("\n")}\n\n" \
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
module PlaiTestr
|
|
2
|
+
module Reporter
|
|
3
|
+
module Duration
|
|
4
|
+
def self.format(seconds)
|
|
5
|
+
seconds = seconds.round
|
|
6
|
+
return "#{seconds / 3600}h #{(seconds % 3600 / 60).to_s.rjust(2, '0')}m" if seconds >= 3600
|
|
7
|
+
|
|
8
|
+
"#{seconds / 60}m #{(seconds % 60).to_s.rjust(2, '0')}s"
|
|
9
|
+
end
|
|
10
|
+
end
|
|
11
|
+
end
|
|
12
|
+
end
|
|
@@ -37,6 +37,7 @@ module PlaiTestr
|
|
|
37
37
|
rows << "| **Personas** | #{data['personas'].map { |p, label| "#{p}: #{label}" }.join('; ')} |" if data['personas']
|
|
38
38
|
rows << "| **Model settings** | #{settings.map { |key, value| "#{key} #{value}" }.join(', ')} |" if settings && !bot_only
|
|
39
39
|
rows << "| **Decisions** | #{@game_state.turn} |"
|
|
40
|
+
rows << "| **Wall time** | #{Duration.format(data['wall_seconds'])} |" if data['wall_seconds']
|
|
40
41
|
rows << "| **Generated** | #{Time.now.strftime('%Y-%m-%d %H:%M')} |"
|
|
41
42
|
|
|
42
43
|
"# Playtest Report: #{@rules.dig('game', 'name')}\n\n| | |\n|---|---|\n#{rows.join("\n")}"
|
|
@@ -62,6 +62,7 @@ module PlaiTestr
|
|
|
62
62
|
@turn_log = []
|
|
63
63
|
@warnings = []
|
|
64
64
|
@running = true
|
|
65
|
+
@started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
65
66
|
@fingerprint = game.fingerprint if persist
|
|
66
67
|
@client = Agents::OllamaClient.new(think_log: (think_log_path if persist), out: out, verbose: verbose)
|
|
67
68
|
@settings = resolve_settings
|
|
@@ -610,11 +611,12 @@ module PlaiTestr
|
|
|
610
611
|
end
|
|
611
612
|
|
|
612
613
|
def finish
|
|
614
|
+
data['wall_seconds'] = data.fetch('wall_seconds', 0) + Process.clock_gettime(Process::CLOCK_MONOTONIC) - @started
|
|
613
615
|
if game_over? && @running
|
|
614
616
|
@status = 'completed'
|
|
615
617
|
save_checkpoint('completed')
|
|
616
618
|
@report_path = generate_report
|
|
617
|
-
out.puts "\nGame complete. Report: #{@report_path}"
|
|
619
|
+
out.puts "\nGame complete in #{Reporter::Duration.format(data['wall_seconds'])}. Report: #{@report_path}"
|
|
618
620
|
else
|
|
619
621
|
@status = 'interrupted'
|
|
620
622
|
save_checkpoint('interrupted')
|
|
@@ -86,7 +86,7 @@ module PlaiTestr
|
|
|
86
86
|
# the event log entries meant for them. Subclasses hide more with
|
|
87
87
|
# `super.except('deck').merge('deck_size' => ...)`. serialize (checkpoints) keeps all of @data.
|
|
88
88
|
def data_for(observer)
|
|
89
|
-
shown = @data.except('warnings')
|
|
89
|
+
shown = @data.except('warnings', 'wall_seconds')
|
|
90
90
|
return shown unless shown.key?('log')
|
|
91
91
|
|
|
92
92
|
shown.merge('log' => EventLog.new(shown['log']).visible_to(observer))
|
data/lib/plai_testr/version.rb
CHANGED
data/lib/plai_testr.rb
CHANGED
|
@@ -43,6 +43,7 @@ require_relative 'plai_testr/agents/referee_agent'
|
|
|
43
43
|
require_relative 'plai_testr/rules_loader'
|
|
44
44
|
require_relative 'plai_testr/personas'
|
|
45
45
|
require_relative 'plai_testr/reporter/tables'
|
|
46
|
+
require_relative 'plai_testr/reporter/duration'
|
|
46
47
|
require_relative 'plai_testr/reporter/think_log'
|
|
47
48
|
require_relative 'plai_testr/reporter/report_builder'
|
|
48
49
|
require_relative 'plai_testr/reporter/model_comparison'
|
|
@@ -60,5 +61,6 @@ require_relative 'plai_testr/deck'
|
|
|
60
61
|
require_relative 'plai_testr/event_log'
|
|
61
62
|
require_relative 'plai_testr/names'
|
|
62
63
|
require_relative 'plai_testr/guide'
|
|
64
|
+
require_relative 'plai_testr/examples'
|
|
63
65
|
require_relative 'plai_testr/generator'
|
|
64
66
|
require_relative 'plai_testr/cli'
|
data/templates/app/AGENTS.md.erb
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
This app playtests tabletop games with [PlAI Testr](https://github.com/armahillo/plai_testr).
|
|
4
4
|
|
|
5
|
-
The PlAI Testr guide for AI assistants is `.plai_testr/gem/docs/AGENTS.md`: read it before working here. `.plai_testr/gem` links to the installed gem (every `plai_testr` command and spec run keeps it current), so its `
|
|
5
|
+
The PlAI Testr guide for AI assistants is `.plai_testr/gem/docs/AGENTS.md`: read it before working here. `.plai_testr/gem` links to the installed gem (every `plai_testr` command and spec run keeps it current), so its `docs/` (including the technical reference in `docs/technical/`) and `templates/examples/` are there too. Gem-wide guidance lives in the gem; this file holds only what is specific to this app.
|
|
6
6
|
|
|
7
7
|
## This app
|
|
8
8
|
|
data/templates/app/README.md.erb
CHANGED
|
@@ -34,7 +34,7 @@ Every command takes `--help` to list its options, e.g. `bundle exec plai_testr h
|
|
|
34
34
|
|
|
35
35
|
`--quiet` shows only a progress line and the result; `--verbose` shows more. Ctrl+C once saves and stops after the current move (a botrun finishes its current game and reports); Ctrl+C twice quits immediately.
|
|
36
36
|
|
|
37
|
-
Settings for the whole app (decision limit, model, Ollama host, context size) are in `config/plai_testr.rb`.
|
|
37
|
+
Settings for the whole app (decision limit, model, Ollama host, context size) are in `config/plai_testr.rb`. [Running playtests](https://github.com/armahillo/plai_testr/blob/main/docs/running-playtests.md) covers every command, option and setting.
|
|
38
38
|
|
|
39
39
|
## Layout
|
|
40
40
|
|
|
@@ -85,9 +85,9 @@ Some ways to start:
|
|
|
85
85
|
bundle exec plai_testr generate game my_game
|
|
86
86
|
```
|
|
87
87
|
|
|
88
|
-
This creates the skeleton above.
|
|
88
|
+
This creates the skeleton above. Write the rules in `data/rules.md` and the components in `data/components/`, then build the harness: see [Ingesting a game with an agent](https://github.com/armahillo/plai_testr/blob/main/docs/ingesting-a-game-with-an-agent.md) or [Ingesting a game manually](https://github.com/armahillo/plai_testr/blob/main/docs/ingesting-a-game-manually.md). The same guides are in `.plai_testr/gem/docs/`, matching the installed version.
|
|
89
89
|
|
|
90
|
-
`bundle exec rspec games/my_game` runs its specs. The
|
|
90
|
+
`bundle exec rspec games/my_game/spec` runs its specs. The standard checks stay pending until the game can play itself to the end.
|
|
91
91
|
|
|
92
92
|
## Removing a game
|
|
93
93
|
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
name,suit,rank
|
|
2
|
+
A♣,clubs,1
|
|
3
|
+
2♣,clubs,2
|
|
4
|
+
3♣,clubs,3
|
|
5
|
+
4♣,clubs,4
|
|
6
|
+
5♣,clubs,5
|
|
7
|
+
6♣,clubs,6
|
|
8
|
+
7♣,clubs,7
|
|
9
|
+
8♣,clubs,8
|
|
10
|
+
9♣,clubs,9
|
|
11
|
+
10♣,clubs,10
|
|
12
|
+
J♣,clubs,11
|
|
13
|
+
Q♣,clubs,12
|
|
14
|
+
K♣,clubs,13
|
|
15
|
+
A♦,diamonds,1
|
|
16
|
+
2♦,diamonds,2
|
|
17
|
+
3♦,diamonds,3
|
|
18
|
+
4♦,diamonds,4
|
|
19
|
+
5♦,diamonds,5
|
|
20
|
+
6♦,diamonds,6
|
|
21
|
+
7♦,diamonds,7
|
|
22
|
+
8♦,diamonds,8
|
|
23
|
+
9♦,diamonds,9
|
|
24
|
+
10♦,diamonds,10
|
|
25
|
+
J♦,diamonds,11
|
|
26
|
+
Q♦,diamonds,12
|
|
27
|
+
K♦,diamonds,13
|
|
28
|
+
A♠,spades,1
|
|
29
|
+
2♠,spades,2
|
|
30
|
+
3♠,spades,3
|
|
31
|
+
4♠,spades,4
|
|
32
|
+
5♠,spades,5
|
|
33
|
+
6♠,spades,6
|
|
34
|
+
7♠,spades,7
|
|
35
|
+
8♠,spades,8
|
|
36
|
+
9♠,spades,9
|
|
37
|
+
10♠,spades,10
|
|
38
|
+
J♠,spades,11
|
|
39
|
+
Q♠,spades,12
|
|
40
|
+
K♠,spades,13
|
|
41
|
+
A♥,hearts,1
|
|
42
|
+
2♥,hearts,2
|
|
43
|
+
3♥,hearts,3
|
|
44
|
+
4♥,hearts,4
|
|
45
|
+
5♥,hearts,5
|
|
46
|
+
6♥,hearts,6
|
|
47
|
+
7♥,hearts,7
|
|
48
|
+
8♥,hearts,8
|
|
49
|
+
9♥,hearts,9
|
|
50
|
+
10♥,hearts,10
|
|
51
|
+
J♥,hearts,11
|
|
52
|
+
Q♥,hearts,12
|
|
53
|
+
K♥,hearts,13
|
|
@@ -9,68 +9,5 @@
|
|
|
9
9
|
"points_to_win": 100,
|
|
10
10
|
"knock_limit": 10,
|
|
11
11
|
"gin_bonus": 25,
|
|
12
|
-
"undercut_bonus": 25
|
|
13
|
-
|
|
14
|
-
"components": [
|
|
15
|
-
{
|
|
16
|
-
"name": "Deck",
|
|
17
|
-
"type": "card_deck",
|
|
18
|
-
"count": 52,
|
|
19
|
-
"description": "Standard 52-card deck. Aces are low (rank 1); face cards count 10.",
|
|
20
|
-
"values": [
|
|
21
|
-
{ "name": "A♣", "rank": 1, "suit": "clubs" },
|
|
22
|
-
{ "name": "2♣", "rank": 2, "suit": "clubs" },
|
|
23
|
-
{ "name": "3♣", "rank": 3, "suit": "clubs" },
|
|
24
|
-
{ "name": "4♣", "rank": 4, "suit": "clubs" },
|
|
25
|
-
{ "name": "5♣", "rank": 5, "suit": "clubs" },
|
|
26
|
-
{ "name": "6♣", "rank": 6, "suit": "clubs" },
|
|
27
|
-
{ "name": "7♣", "rank": 7, "suit": "clubs" },
|
|
28
|
-
{ "name": "8♣", "rank": 8, "suit": "clubs" },
|
|
29
|
-
{ "name": "9♣", "rank": 9, "suit": "clubs" },
|
|
30
|
-
{ "name": "10♣", "rank": 10, "suit": "clubs" },
|
|
31
|
-
{ "name": "J♣", "rank": 11, "suit": "clubs" },
|
|
32
|
-
{ "name": "Q♣", "rank": 12, "suit": "clubs" },
|
|
33
|
-
{ "name": "K♣", "rank": 13, "suit": "clubs" },
|
|
34
|
-
{ "name": "A♦", "rank": 1, "suit": "diamonds" },
|
|
35
|
-
{ "name": "2♦", "rank": 2, "suit": "diamonds" },
|
|
36
|
-
{ "name": "3♦", "rank": 3, "suit": "diamonds" },
|
|
37
|
-
{ "name": "4♦", "rank": 4, "suit": "diamonds" },
|
|
38
|
-
{ "name": "5♦", "rank": 5, "suit": "diamonds" },
|
|
39
|
-
{ "name": "6♦", "rank": 6, "suit": "diamonds" },
|
|
40
|
-
{ "name": "7♦", "rank": 7, "suit": "diamonds" },
|
|
41
|
-
{ "name": "8♦", "rank": 8, "suit": "diamonds" },
|
|
42
|
-
{ "name": "9♦", "rank": 9, "suit": "diamonds" },
|
|
43
|
-
{ "name": "10♦", "rank": 10, "suit": "diamonds" },
|
|
44
|
-
{ "name": "J♦", "rank": 11, "suit": "diamonds" },
|
|
45
|
-
{ "name": "Q♦", "rank": 12, "suit": "diamonds" },
|
|
46
|
-
{ "name": "K♦", "rank": 13, "suit": "diamonds" },
|
|
47
|
-
{ "name": "A♠", "rank": 1, "suit": "spades" },
|
|
48
|
-
{ "name": "2♠", "rank": 2, "suit": "spades" },
|
|
49
|
-
{ "name": "3♠", "rank": 3, "suit": "spades" },
|
|
50
|
-
{ "name": "4♠", "rank": 4, "suit": "spades" },
|
|
51
|
-
{ "name": "5♠", "rank": 5, "suit": "spades" },
|
|
52
|
-
{ "name": "6♠", "rank": 6, "suit": "spades" },
|
|
53
|
-
{ "name": "7♠", "rank": 7, "suit": "spades" },
|
|
54
|
-
{ "name": "8♠", "rank": 8, "suit": "spades" },
|
|
55
|
-
{ "name": "9♠", "rank": 9, "suit": "spades" },
|
|
56
|
-
{ "name": "10♠", "rank": 10, "suit": "spades" },
|
|
57
|
-
{ "name": "J♠", "rank": 11, "suit": "spades" },
|
|
58
|
-
{ "name": "Q♠", "rank": 12, "suit": "spades" },
|
|
59
|
-
{ "name": "K♠", "rank": 13, "suit": "spades" },
|
|
60
|
-
{ "name": "A♥", "rank": 1, "suit": "hearts" },
|
|
61
|
-
{ "name": "2♥", "rank": 2, "suit": "hearts" },
|
|
62
|
-
{ "name": "3♥", "rank": 3, "suit": "hearts" },
|
|
63
|
-
{ "name": "4♥", "rank": 4, "suit": "hearts" },
|
|
64
|
-
{ "name": "5♥", "rank": 5, "suit": "hearts" },
|
|
65
|
-
{ "name": "6♥", "rank": 6, "suit": "hearts" },
|
|
66
|
-
{ "name": "7♥", "rank": 7, "suit": "hearts" },
|
|
67
|
-
{ "name": "8♥", "rank": 8, "suit": "hearts" },
|
|
68
|
-
{ "name": "9♥", "rank": 9, "suit": "hearts" },
|
|
69
|
-
{ "name": "10♥", "rank": 10, "suit": "hearts" },
|
|
70
|
-
{ "name": "J♥", "rank": 11, "suit": "hearts" },
|
|
71
|
-
{ "name": "Q♥", "rank": 12, "suit": "hearts" },
|
|
72
|
-
{ "name": "K♥", "rank": 13, "suit": "hearts" }
|
|
73
|
-
]
|
|
74
|
-
}
|
|
75
|
-
]
|
|
12
|
+
"undercut_bonus": 25
|
|
76
13
|
}
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: plai_testr
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.2.
|
|
4
|
+
version: 0.2.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Aaron H
|
|
@@ -68,8 +68,20 @@ files:
|
|
|
68
68
|
- README.md
|
|
69
69
|
- bin/plai_testr
|
|
70
70
|
- docs/AGENTS.md
|
|
71
|
-
- docs/
|
|
72
|
-
- docs/
|
|
71
|
+
- docs/getting-started.md
|
|
72
|
+
- docs/ingesting-a-game-manually.md
|
|
73
|
+
- docs/ingesting-a-game-with-an-agent.md
|
|
74
|
+
- docs/running-playtests.md
|
|
75
|
+
- docs/technical/README.md
|
|
76
|
+
- docs/technical/debrief.md
|
|
77
|
+
- docs/technical/frame-queue.md
|
|
78
|
+
- docs/technical/personas-and-bots.md
|
|
79
|
+
- docs/technical/referee.md
|
|
80
|
+
- docs/technical/reports.md
|
|
81
|
+
- docs/technical/rules-and-components.md
|
|
82
|
+
- docs/technical/runner.md
|
|
83
|
+
- docs/technical/specs.md
|
|
84
|
+
- docs/technical/state.md
|
|
73
85
|
- lib/plai_testr.rb
|
|
74
86
|
- lib/plai_testr/agents/ollama_client.rb
|
|
75
87
|
- lib/plai_testr/agents/player_agent.rb
|
|
@@ -80,6 +92,7 @@ files:
|
|
|
80
92
|
- lib/plai_testr/deck.rb
|
|
81
93
|
- lib/plai_testr/errors.rb
|
|
82
94
|
- lib/plai_testr/event_log.rb
|
|
95
|
+
- lib/plai_testr/examples.rb
|
|
83
96
|
- lib/plai_testr/game.rb
|
|
84
97
|
- lib/plai_testr/generator.rb
|
|
85
98
|
- lib/plai_testr/guide.rb
|
|
@@ -88,6 +101,7 @@ files:
|
|
|
88
101
|
- lib/plai_testr/personas.rb
|
|
89
102
|
- lib/plai_testr/preflight.rb
|
|
90
103
|
- lib/plai_testr/reporter/botrun_report.rb
|
|
104
|
+
- lib/plai_testr/reporter/duration.rb
|
|
91
105
|
- lib/plai_testr/reporter/model_comparison.rb
|
|
92
106
|
- lib/plai_testr/reporter/report_builder.rb
|
|
93
107
|
- lib/plai_testr/reporter/tables.rb
|
|
@@ -111,6 +125,7 @@ files:
|
|
|
111
125
|
- templates/app/spec/spec_helper.rb
|
|
112
126
|
- templates/examples/gin_rummy/NOTES.md
|
|
113
127
|
- templates/examples/gin_rummy/data/bots.yml
|
|
128
|
+
- templates/examples/gin_rummy/data/components/cards.csv
|
|
114
129
|
- templates/examples/gin_rummy/data/personas.yml
|
|
115
130
|
- templates/examples/gin_rummy/data/rules.json
|
|
116
131
|
- templates/examples/gin_rummy/data/rules.md
|
|
File without changes
|
|
File without changes
|