alfrd-gsheet 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- alfrd_gsheet-0.2.0/PKG-INFO +530 -0
- alfrd_gsheet-0.2.0/README.md +522 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/__init__.py +33 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/a1.py +63 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/actions.py +454 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/cli.py +274 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/client.py +348 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/columns.py +94 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/loopback.py +214 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/mapping.py +380 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/oauth.py +362 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/oauth_client.json +1 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/pages/picker.html +35 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/pages/signed_in.html +2 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/project.py +260 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/row_match.py +43 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/settings.py +144 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/studio.py +119 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/sync.py +505 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/web/index.css +106 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/web/index.js +88 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet/web/project_ui.js +959 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet.egg-info/PKG-INFO +530 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet.egg-info/SOURCES.txt +38 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet.egg-info/dependency_links.txt +1 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet.egg-info/entry_points.txt +2 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet.egg-info/requires.txt +2 -0
- alfrd_gsheet-0.2.0/alfrd_gsheet.egg-info/top_level.txt +1 -0
- alfrd_gsheet-0.2.0/pyproject.toml +20 -0
- alfrd_gsheet-0.2.0/setup.cfg +4 -0
- alfrd_gsheet-0.2.0/tests/test_client.py +380 -0
- alfrd_gsheet-0.2.0/tests/test_core.py +537 -0
- alfrd_gsheet-0.2.0/tests/test_gsheet_actions.py +536 -0
- alfrd_gsheet-0.2.0/tests/test_gsheet_cli.py +209 -0
- alfrd_gsheet-0.2.0/tests/test_gsheet_studio.py +188 -0
- alfrd_gsheet-0.2.0/tests/test_loopback.py +119 -0
- alfrd_gsheet-0.2.0/tests/test_oauth.py +278 -0
- alfrd_gsheet-0.2.0/tests/test_oauth_config.py +248 -0
- alfrd_gsheet-0.2.0/tests/test_real_auth.py +51 -0
- alfrd_gsheet-0.2.0/tests/test_runner.py +108 -0
|
@@ -0,0 +1,530 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: alfrd-gsheet
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Requires-Python: >=3.10
|
|
5
|
+
Description-Content-Type: text/markdown
|
|
6
|
+
Requires-Dist: google-auth[requests]==2.61.0
|
|
7
|
+
Requires-Dist: requests==2.32.5
|
|
8
|
+
|
|
9
|
+
# Google Sheet step sync
|
|
10
|
+
|
|
11
|
+
Writes each finished plan step's status, timing, usage and other fields into
|
|
12
|
+
a Google Sheet row for that target. It can also read notes or todo/skip
|
|
13
|
+
requests from the sheet into the plan. Mirroring runs in the plan runner's step
|
|
14
|
+
hooks. The plugin also provides the `alfrd gsheet` CLI, a Studio panel and a
|
|
15
|
+
palette check.
|
|
16
|
+
|
|
17
|
+
Status: offline transport tests, the scheduler integration, an isolated
|
|
18
|
+
install and the Studio browser tests pass. Live verification against a real
|
|
19
|
+
Google Sheet is still pending.
|
|
20
|
+
Installing the plugin registers hooks; projects without a mapping make no API
|
|
21
|
+
calls and do not need credentials.
|
|
22
|
+
|
|
23
|
+
## Setup walkthrough: Google sign-in (desktop)
|
|
24
|
+
|
|
25
|
+
1. Install the plugin and restart `alfrd serve`:
|
|
26
|
+
`alfrd plugin install ./examples/plugins/alfrd-gsheet`.
|
|
27
|
+
2. Open a project's **Google Sheet → Connect & validate…** dialog and choose
|
|
28
|
+
**Sign in with Google…**. Complete Google's consent screen in the browser on
|
|
29
|
+
the same computer as ALFRD. Select `oauth` in **Settings → Plugins → Google
|
|
30
|
+
Sheet → Sign-in method** if you previously used a service-account key.
|
|
31
|
+
3. **Connect sheet… → Choose from Google Drive…**, or enter a title and
|
|
32
|
+
**Create new spreadsheet**. Picker grants ALFRD access to the chosen file.
|
|
33
|
+
Pasting an arbitrary spreadsheet URL alone cannot grant access. Creating a
|
|
34
|
+
sheet starts it empty: open its link, add headers and target rows, then Load.
|
|
35
|
+
4. Select the tab, header row and row-match column, then **Attach**. Validate
|
|
36
|
+
the mapping and preview an export before writing run results.
|
|
37
|
+
|
|
38
|
+
Google sign-in requests only `drive.file`, `openid` and email. It can read and
|
|
39
|
+
edit files chosen with Picker or created by ALFRD, rather than all Drive files.
|
|
40
|
+
**Choose sheet again…** restores a file grant without changing the mapping.
|
|
41
|
+
A read-only sheet needs Editor permission from its owner. Expired or revoked
|
|
42
|
+
sign-in stops subsequent sync requests until you sign in again. The local panel
|
|
43
|
+
shows a sign-in or file-access recovery message after a failed sync.
|
|
44
|
+
|
|
45
|
+
**Sign out…** attempts to revoke the grant, then deletes the local refresh token
|
|
46
|
+
whether or not Google is reachable. Google's revocation affects **all clients
|
|
47
|
+
of the same Google project**, including ALFRD on other computers. Previously
|
|
48
|
+
chosen files may need choosing again. If revoke fails, remove ALFRD manually at
|
|
49
|
+
<https://myaccount.google.com/connections>.
|
|
50
|
+
|
|
51
|
+
CLI equivalents:
|
|
52
|
+
|
|
53
|
+
```sh
|
|
54
|
+
alfrd gsheet auth login
|
|
55
|
+
alfrd gsheet auth status
|
|
56
|
+
alfrd gsheet auth logout --yes
|
|
57
|
+
alfrd gsheet create --title "Lab runs"
|
|
58
|
+
alfrd gsheet init <project> --spreadsheet <returned-id>
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
`auth login --no-browser --timeout 300` prints a URL and waits; advanced remote
|
|
62
|
+
users must forward the printed loopback port themselves. Normal login refuses
|
|
63
|
+
headless/SSH sessions and starts no listener there. Servers and CI should use
|
|
64
|
+
**service_account**, below. Hooks never initiate login.
|
|
65
|
+
|
|
66
|
+
### OAuth client configuration
|
|
67
|
+
|
|
68
|
+
**End users of the official ALFRD release need no OAuth client setup or credentials.**
|
|
69
|
+
The release CI supplies the bundled client; users just choose **Sign in with Google**.
|
|
70
|
+
|
|
71
|
+
The repository ships **empty placeholders** in `alfrd_gsheet/oauth_client.json`.
|
|
72
|
+
For source checkouts, maintainers can save their client outside the repository at
|
|
73
|
+
`~/.config/alfrd/gsheet_oauth_client.json` (or
|
|
74
|
+
`$XDG_CONFIG_HOME/alfrd/gsheet_oauth_client.json` when set). Discovery is automatic.
|
|
75
|
+
Use Google's downloaded Desktop `{"installed": {...}}` JSON or a flat object
|
|
76
|
+
with the keys below. Optional `picker_api_key` and `project_number` keys can be
|
|
77
|
+
included in the same object as the client keys; `project_id` is not the project number.
|
|
78
|
+
Web clients are rejected. Use `chmod 600` on the file; group- or world-readable
|
|
79
|
+
client files trigger a warning without preventing sign-in.
|
|
80
|
+
|
|
81
|
+
Precedence is **per key**, highest first: non-empty individual environment
|
|
82
|
+
variables → `ALFRD_GSHEET_OAUTH_CLIENT_FILE` → per-user file → bundled defaults.
|
|
83
|
+
The explicit file variable accepts the same JSON formats. Missing per-user files
|
|
84
|
+
are ignored; malformed or unreadable files produce a redacted configuration error.
|
|
85
|
+
Set overrides in both the Studio server and runner/CLI environments.
|
|
86
|
+
Maintainer setup hints are logged by the server when the client is unavailable;
|
|
87
|
+
Studio directs users to the official release or **Advanced → service account**.
|
|
88
|
+
See the [maintainer setup guide](../../../docs/plugins/gsheet-oauth-setup.md).
|
|
89
|
+
|
|
90
|
+
| File key | Environment override | Format |
|
|
91
|
+
|---|---|---|
|
|
92
|
+
| `client_id` | `ALFRD_GSHEET_OAUTH_CLIENT_ID` | Desktop client ending in `.apps.googleusercontent.com` |
|
|
93
|
+
| `client_secret` | `ALFRD_GSHEET_OAUTH_CLIENT_SECRET` | Non-empty desktop client secret, at most 256 characters |
|
|
94
|
+
| `picker_api_key` | `ALFRD_GSHEET_PICKER_API_KEY` | API key, 30–64 letters/digits/underscore/hyphen |
|
|
95
|
+
| `project_number` | `ALFRD_GSHEET_GCP_PROJECT_NUMBER` | Numeric Cloud project number, 6–20 digits |
|
|
96
|
+
|
|
97
|
+
Use one Google Cloud project for the Desktop OAuth client and Picker project
|
|
98
|
+
number. Enable Sheets, Drive and Google Picker APIs. Restrict the API key to
|
|
99
|
+
the Google Picker API. Loopback origins use random ports; validate any website
|
|
100
|
+
restrictions before distributing your build. Without Picker configuration,
|
|
101
|
+
sign-in and Create work; choosing existing files is disabled.
|
|
102
|
+
|
|
103
|
+
Configure the external consent screen for the three scopes above and publish
|
|
104
|
+
it **In production** before release. In **Testing**, grants containing
|
|
105
|
+
`drive.file` normally expire after **7 days**. Complete applicable Google brand
|
|
106
|
+
verification. No real client credentials, API keys or user tokens belong in
|
|
107
|
+
source control.
|
|
108
|
+
|
|
109
|
+
Refresh tokens live separately from plugin settings in
|
|
110
|
+
`<user_config_dir("alfrd")>/gsheet/oauth-token.json` (directory 0700, file 0600,
|
|
111
|
+
atomic replacement, honours `XDG_CONFIG_HOME`). Access tokens remain in process
|
|
112
|
+
memory and the isolated Picker page; they are never returned in Studio actions.
|
|
113
|
+
This is local file protection, not an OS keyring. Only short-lived nonces and
|
|
114
|
+
URLs pass between Studio and the `127.0.0.1:<random-port>` helper. The helper
|
|
115
|
+
checks Host, state/PKCE and single-use callbacks, and expires automatically.
|
|
116
|
+
|
|
117
|
+
**Release validation still needed:** the brief's S0 checks A1–A4 require a
|
|
118
|
+
maintainer Google project/account: Desktop-token Picker on the random loopback
|
|
119
|
+
origin, API-key restrictions/CSP, inaccessible-file 403/404, and re-picking after
|
|
120
|
+
revocation. These assumptions have not been live-verified by the offline tests.
|
|
121
|
+
If Desktop-token web Picker is rejected, the brief describes a separate
|
|
122
|
+
`trigger_onepick` desktop fallback; this build implements the web Picker path.
|
|
123
|
+
See [Google installed-app OAuth](https://developers.google.com/identity/protocols/oauth2/native-app)
|
|
124
|
+
and [Google Picker integration](https://developers.google.com/workspace/drive/picker/guides/web-picker).
|
|
125
|
+
|
|
126
|
+
## Advanced: service account (headless)
|
|
127
|
+
|
|
128
|
+
1. **Install** the plugin and restart `alfrd serve`:
|
|
129
|
+
`alfrd plugin install ./examples/plugins/alfrd-gsheet`.
|
|
130
|
+
2. **Create a service account** in a Google Cloud project (IAM & Admin →
|
|
131
|
+
Service accounts), add a JSON key, and download it. Enable the
|
|
132
|
+
**Google Sheets API** in the same project. The **Google Drive API** is only
|
|
133
|
+
needed if you tick **Check edit permission with the Drive API**.
|
|
134
|
+
3. **Share the spreadsheet** with the key's `client_email`
|
|
135
|
+
(`…@….iam.gserviceaccount.com`) as an **Editor**. Use a dedicated sheet or
|
|
136
|
+
a copy until you trust the mapping.
|
|
137
|
+
4. **Configure**: in Settings → Plugins → Google Sheet → Configure, select
|
|
138
|
+
`service_account` as **Sign-in method**, paste the
|
|
139
|
+
whole JSON key and set the default spreadsheet id or URL. Press **Test**. It
|
|
140
|
+
writes nothing; it reports Editor access and the number of worksheets.
|
|
141
|
+
5. **Generate a mapping**:
|
|
142
|
+
`alfrd gsheet init <project> [--worksheet Targets]`. The sheet needs a header
|
|
143
|
+
row and a column whose values match the plan's target names.
|
|
144
|
+
Edit the file to add fields, formats or inbound rules (see below).
|
|
145
|
+
6. **Validate**: `alfrd gsheet validate <project>`, or **Google Sheet: validate
|
|
146
|
+
mapping** in the Studio palette. Fix each reported column or step.
|
|
147
|
+
7. **Preview, then backfill**: `alfrd gsheet push <project> --dry-run` shows the
|
|
148
|
+
cells it would write. Run without `--dry-run` to fill in steps that finished
|
|
149
|
+
before the mapping existed. From then on, every launched step syncs on its
|
|
150
|
+
own.
|
|
151
|
+
8. Optionally add the [Studio panel](#studio-panel-and-validation).
|
|
152
|
+
|
|
153
|
+
To stop syncing a project, set `enabled: false` or delete `alfrd.gsheet.yaml`.
|
|
154
|
+
The setting applies from the next launched unit.
|
|
155
|
+
|
|
156
|
+
## Development
|
|
157
|
+
|
|
158
|
+
From the alfrd repository, run the offline tests with the host environment:
|
|
159
|
+
|
|
160
|
+
```sh
|
|
161
|
+
uv run python -m pytest -q examples/plugins/alfrd-gsheet/tests
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
No service account, Google libraries or network are needed to run these tests.
|
|
165
|
+
The optional real-library signing and OAuth refresh tests skip when Google's libraries are absent.
|
|
166
|
+
To include them without accessing a real sheet:
|
|
167
|
+
|
|
168
|
+
```sh
|
|
169
|
+
uv run --with 'google-auth[requests]==2.61.0' --with 'requests==2.32.5' \
|
|
170
|
+
python -m pytest -q examples/plugins/alfrd-gsheet/tests
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
The suite blocks non-loopback socket connections. Loopback HTTP handler tests
|
|
174
|
+
use in-memory connections and fake Google responses, so even restricted CI
|
|
175
|
+
needs no listening sockets. Browser-origin/Google interoperability remains a
|
|
176
|
+
manual release check; JS syntax is checked with `node --check`.
|
|
177
|
+
|
|
178
|
+
The engine accepts a `SheetsClient`; its protocol is in `alfrd_gsheet/client.py`.
|
|
179
|
+
The transport dependencies are pinned in `pyproject.toml`; alfrd and its
|
|
180
|
+
PyYAML/typer dependencies come from the host, not a second alfrd install.
|
|
181
|
+
|
|
182
|
+
Install from the repository:
|
|
183
|
+
|
|
184
|
+
```sh
|
|
185
|
+
alfrd plugin install ./examples/plugins/alfrd-gsheet
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
## Settings and Test
|
|
189
|
+
|
|
190
|
+
**Sign-in method** accepts `oauth` or `service_account`. Empty/missing uses
|
|
191
|
+
`service_account` when `credentials_json` is present, otherwise `oauth`. Existing
|
|
192
|
+
key-only configurations need no migration or file rewrite. Explicit `oauth`
|
|
193
|
+
ignores a leftover key without deleting it; switching back restores that key.
|
|
194
|
+
OAuth can be saved without any service-account key. Without a default sheet,
|
|
195
|
+
OAuth Test refreshes sign-in and reports the account; set a default to also test
|
|
196
|
+
sheet access.
|
|
197
|
+
|
|
198
|
+
For Advanced mode, select `service_account`, paste a service-account JSON
|
|
199
|
+
key into **Service-account key JSON (Advanced)** and set **Default spreadsheet id or URL**.
|
|
200
|
+
Enable the Sheets API in the key's Google Cloud project and share the
|
|
201
|
+
spreadsheet with the key's `client_email` as an Editor. Credentials stay
|
|
202
|
+
in the host's private plugin settings file; public settings show only whether
|
|
203
|
+
the key is set. Do not put the key in the project mapping.
|
|
204
|
+
|
|
205
|
+
**Test** reads spreadsheet metadata and lists worksheets without writing; a
|
|
206
|
+
successful result reports the worksheet count. By default it checks read access
|
|
207
|
+
only, and edit access is verified on the first write. Tick **Check edit
|
|
208
|
+
permission with the Drive API** to have Test also ask Drive whether the account
|
|
209
|
+
can edit; that needs the Drive API enabled in the same project. Syncing itself
|
|
210
|
+
uses only the Sheets API.
|
|
211
|
+
Read-only access asks you to grant Editor permission; 403/404 errors ask you to
|
|
212
|
+
check sharing, the spreadsheet id and enabled APIs. In service-account mode, Test requires a default
|
|
213
|
+
spreadsheet even when individual mappings specify their own ids.
|
|
214
|
+
|
|
215
|
+
Defaults: `value_input_option: RAW`, `conflict_policy: skip`,
|
|
216
|
+
`request_timeout: 15`, `dry_run: false`, `check_drive: false`. `USER_ENTERED` lets Sheets interpret
|
|
217
|
+
values as formulas/numbers/dates. Request timeout must be positive and finite.
|
|
218
|
+
Credentials are cached per process, while each API call uses a fresh session.
|
|
219
|
+
Authentication, lock waits, up to three attempts for 429/5xx, and exponential
|
|
220
|
+
backoff with jitter share the caller's absolute deadline. Socket timeouts are
|
|
221
|
+
capped by the request timeout and remaining budget. A wall-time guard also
|
|
222
|
+
bounds authentication/library waits; an already-sent HTTP write may finish
|
|
223
|
+
after timeout, so a timeout does not prove the write was rejected. No subsequent
|
|
224
|
+
retry or API request starts after the deadline.
|
|
225
|
+
|
|
226
|
+
## Mapping file
|
|
227
|
+
|
|
228
|
+
Save `<project>/alfrd.gsheet.yaml`. This file is separate from `alfrd.yaml` and
|
|
229
|
+
is re-read before each launched unit. A unit keeps the mapping that it started
|
|
230
|
+
with. Missing files, `enabled: false`, and units with no matching rules do
|
|
231
|
+
nothing. Skipped, blocked and never-launched units get no lifecycle calls.
|
|
232
|
+
|
|
233
|
+
```yaml
|
|
234
|
+
version: 1
|
|
235
|
+
enabled: true
|
|
236
|
+
spreadsheet_id: 1AbCdEfGhIjKlMnOpQrStUvWxYz
|
|
237
|
+
worksheet: Targets # or gid: 0 instead of worksheet
|
|
238
|
+
header_row: 1
|
|
239
|
+
rows:
|
|
240
|
+
key_column: TARGET_NAME
|
|
241
|
+
code_column: PROJECT_CODE # optional; distinguishes repeated targets
|
|
242
|
+
missing_row: skip # skip | append
|
|
243
|
+
read_range: A1:ZZ # optional; must include header_row
|
|
244
|
+
verify_before_write: true
|
|
245
|
+
outbound:
|
|
246
|
+
- step: calibrate
|
|
247
|
+
column: calibrate
|
|
248
|
+
field: status
|
|
249
|
+
- step: calibrate
|
|
250
|
+
column: calibrate RAM
|
|
251
|
+
field: usage.peak_mem
|
|
252
|
+
format: bytes
|
|
253
|
+
- step: '*'
|
|
254
|
+
column: '{step} finished'
|
|
255
|
+
field: finished
|
|
256
|
+
format: datetime
|
|
257
|
+
when: [done, failed]
|
|
258
|
+
- step: calibrate
|
|
259
|
+
column: calibrate summary
|
|
260
|
+
field: template
|
|
261
|
+
template: '{status} ({duration_s:.0f}s)'
|
|
262
|
+
inbound:
|
|
263
|
+
- column: notes
|
|
264
|
+
to: plan_column
|
|
265
|
+
plan_column: notes
|
|
266
|
+
- column: image requested
|
|
267
|
+
to: plan_cell
|
|
268
|
+
step: image
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
Headers match after trimming whitespace and ignoring case. A matching header
|
|
272
|
+
takes precedence over a literal column letter, including letters beyond Z.
|
|
273
|
+
Ambiguous headers and repeated row identities fail validation. Outbound rules
|
|
274
|
+
cannot change the target/code columns. Wildcards expand over project steps;
|
|
275
|
+
every substituted header must exist. Duplicate `(step, column)` targets are
|
|
276
|
+
rejected, including duplicates expressed once as a header and once as letters.
|
|
277
|
+
|
|
278
|
+
Inbound `plan_column` must name an existing extra column. It cannot change a
|
|
279
|
+
workflow step or the target, code, workdir or files columns. Inbound `plan_cell`
|
|
280
|
+
accepts only `todo` and `skip`. It calls the host's CSV updater under the plan
|
|
281
|
+
lock with an `only_if` guard excluding `running`. No step in the about-to-run
|
|
282
|
+
unit is touched; inbound requests affect later steps. Dry run changes neither
|
|
283
|
+
the sheet nor the plan CSV.
|
|
284
|
+
|
|
285
|
+
### Matching rows by filenames
|
|
286
|
+
|
|
287
|
+
Existing mappings default to target-name matching. To match an AVICA sheet's
|
|
288
|
+
`FILENAMES` column, choose **Row-match column → FILENAMES** and **Match against →
|
|
289
|
+
Plan filenames** in Connect sheet, or edit Sync options. The persisted setting is:
|
|
290
|
+
|
|
291
|
+
```yaml
|
|
292
|
+
rows:
|
|
293
|
+
key_column: FILENAMES
|
|
294
|
+
match_against: files
|
|
295
|
+
# code_column: PROJECT_CODE # optional disambiguation
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
The backend resolves sheet values to the plan's targets; target names in the plan
|
|
299
|
+
are preserved. Commas, whitespace and newlines separate filenames, as in ALFRD's
|
|
300
|
+
plan CSV. Duplicate filenames and list order do not matter. Relative path segments
|
|
301
|
+
are normalized (`./a.fits` matches `a.fits`); matching is case-sensitive and does
|
|
302
|
+
not guess basenames or depend on files being present on disk. Every filename in a
|
|
303
|
+
sheet cell must belong to the same plan target. A cell can contain one file from
|
|
304
|
+
a target that has several files.
|
|
305
|
+
|
|
306
|
+
Setup shows backend-resolved samples (`sheet value → target`). Unknown file lists
|
|
307
|
+
are left unmatched; ambiguous lists prevent attachment/validation/sync. Two sheet
|
|
308
|
+
rows resolving to the same target also prevent sync. A configured project-code
|
|
309
|
+
column disambiguates shared filenames. If missing-row append is enabled, the new
|
|
310
|
+
row uses the plan's complete file list, never its target name; empty or ambiguous
|
|
311
|
+
file lists cannot be appended. Hooks read the running plan CSV; Studio validation
|
|
312
|
+
and connection samples read the configured project plan CSV.
|
|
313
|
+
|
|
314
|
+
### Creating missing destination columns
|
|
315
|
+
|
|
316
|
+
When a named outbound destination is missing, the editor lists its concrete names
|
|
317
|
+
(including each `{step}` expansion). **Review column creation…** shows the worksheet,
|
|
318
|
+
header row and destination letters. Confirm **Create N columns** to insert new
|
|
319
|
+
columns and write their headers. Existing headers/data are kept; your draft stays
|
|
320
|
+
open, and **Save** remains a separate action. “Add status columns for all steps”
|
|
321
|
+
continues to add mapping rules for existing headers only.
|
|
322
|
+
|
|
323
|
+
New columns are inserted at the sheet's actual grid end, after its existing empty
|
|
324
|
+
columns too. Widen or clear a bounded read range if the reviewed letters are outside
|
|
325
|
+
it. Missing row keys, inbound sources, duplicate headers and literal column letters
|
|
326
|
+
need manual correction. Review rechecks current headers, and stale reviews cannot
|
|
327
|
+
write. After a failure, review again to reconcile any completed write before retrying;
|
|
328
|
+
insertion requests are never automatically replayed. Settings dry-run offers
|
|
329
|
+
**Preview column creation** and writes nothing.
|
|
330
|
+
|
|
331
|
+
## Fields and formats
|
|
332
|
+
|
|
333
|
+
Fields: `status`, `exit_code`, `error`, `started`, `finished`, `duration_s`,
|
|
334
|
+
`log_path`, `log_path_rel`, `total_cost_usd`, `model`, `outcome`, `unit_id`,
|
|
335
|
+
`plan_id`, `target`, `code`, `usage.<key>`, `agent_usage.<key>`, `results.<key>`,
|
|
336
|
+
and `template`. Status comes from the individual row/step cell, falling back to
|
|
337
|
+
the unit status when unavailable. Agent/results dotted paths may be nested.
|
|
338
|
+
|
|
339
|
+
Usage keys: `wall_s`, `cpu_s`, `avg_cores`, `peak_mem`, `peak_mem_kind`,
|
|
340
|
+
`peak_rss`, `max_rss`, `read_bytes`, `write_bytes`, `samples`, `interval_s`,
|
|
341
|
+
`limited`, `counted`, `cgroup_memory_peak`, `workdir`, `workdir_start`,
|
|
342
|
+
`workdir_end`. Compatibility aliases: `wall` → `wall_s`, `cpu` → `cpu_s`,
|
|
343
|
+
`cores` → `avg_cores`, `io_read` → `read_bytes`, `io_write` → `write_bytes`.
|
|
344
|
+
|
|
345
|
+
| Format | Example |
|
|
346
|
+
| --- | --- |
|
|
347
|
+
| `raw` (default) | `done`, `0`; None/NaN becomes empty |
|
|
348
|
+
| `bytes` | `1048576` → `1 MiB` |
|
|
349
|
+
| `seconds` | `12.5` → `12.5s` |
|
|
350
|
+
| `duration` | `3661` → `01:01:01` |
|
|
351
|
+
| `datetime` | ISO timestamp (retains timezone) |
|
|
352
|
+
| `percent` | `0.125` → `12.5%` |
|
|
353
|
+
| `json` | Compact JSON with sorted object keys |
|
|
354
|
+
|
|
355
|
+
Templates use the same fields and Python format specifications. For example,
|
|
356
|
+
`{status} ({duration_s:.0f}s)`. Missing values are empty; a numeric specification
|
|
357
|
+
on a missing value is a validation/formatting error and suppresses that sync.
|
|
358
|
+
|
|
359
|
+
## Writes and conflicts
|
|
360
|
+
|
|
361
|
+
Ordinary units read once before launch. After completion, an empty diff makes
|
|
362
|
+
no further API calls. Changed cells are checked in one `batchGet` (unless
|
|
363
|
+
`verify_before_write: false`) and sent in one `batchUpdate`, each range naming
|
|
364
|
+
one cell. Appending a missing target takes an additional full used-range read
|
|
365
|
+
under the sheet lock to choose vacant rows safely. Existing rows retain their
|
|
366
|
+
original snapshot for conflict checks.
|
|
367
|
+
|
|
368
|
+
Policies: `skip` omits cells edited since the snapshot; `overwrite` writes
|
|
369
|
+
anyway; `fail-soft` writes nothing for that unit and reports a hook warning.
|
|
370
|
+
Append refuses a row whose identity or header changed before writing, even
|
|
371
|
+
under `overwrite`. Re-adopted units take a fresh snapshot and do not replay
|
|
372
|
+
inbound changes.
|
|
373
|
+
|
|
374
|
+
The engine has a 25-second total budget per call, leaving margin for the
|
|
375
|
+
30-second public hook timeout. All transport methods receive the same absolute
|
|
376
|
+
deadline, and both sheet and plan lock acquisition are bounded. POSIX flock
|
|
377
|
+
serializes processes under `<project>/.alfrd/locks/`; on Windows the sheet lock
|
|
378
|
+
serializes threads in this process only, matching the host's flock limitation.
|
|
379
|
+
|
|
380
|
+
History is written to `.alfrd/gsheet/sync.jsonl`, one entry per outcome or failed
|
|
381
|
+
before call. It includes changed cell addresses, counts, conflicts and safe
|
|
382
|
+
error text. Values, settings and context payloads are omitted to keep secrets
|
|
383
|
+
out of history. The core reports failures in `runner.log` and
|
|
384
|
+
`plugin.hook_failed` without changing the unit's status.
|
|
385
|
+
|
|
386
|
+
## Command line
|
|
387
|
+
|
|
388
|
+
All commands take the project folder. Errors print one `gsheet: …` line on
|
|
389
|
+
stderr and exit 1. Credentials and Google response bodies are never printed.
|
|
390
|
+
|
|
391
|
+
| Command | What it does |
|
|
392
|
+
|---|---|
|
|
393
|
+
| `alfrd gsheet init <project> [--spreadsheet ID\|URL] [--worksheet NAME] [--header-row N] [--force]` | Reads the header row (default: first tab, row 1) and writes a starter `alfrd.gsheet.yaml`. Each step id that matches a column name gets a `status` rule. The file also includes commented usage/time/log examples. Without `--spreadsheet`, the default spreadsheet from Settings is used and not written to the file. The command refuses to overwrite the file without `--force`. |
|
|
394
|
+
| `alfrd gsheet validate <project> [--offline]` | Checks the mapping against the project's steps, then against the live header (one read). `--offline` skips the sheet. A missing or disabled mapping reports "Sync is off" and exits 0. |
|
|
395
|
+
| `alfrd gsheet diff <project> --unit ID [--plan ID]` | Replays one finished unit's after-sync against the current sheet. Prints `address: 'old' → 'new'` for each cell that would change. Writes nothing, including history. |
|
|
396
|
+
| `alfrd gsheet push <project> [--targets T1,T2] [--steps s1,s2] [--plan ID] [--dry-run]` | Backfills from finished units in all plans (or `--plan`). It takes the newest unit per target and step and uses the same diff and conflict policy as the hooks: one read, one conflict read, one write. The outcome is recorded in history as unit `push`. The `dry_run` setting also turns a push into a preview. |
|
|
397
|
+
| `alfrd gsheet status <project> [-n 20] [--json]` | The most recent `sync.jsonl` entries (UTC time, unit, steps, result, cells, conflicts, error). |
|
|
398
|
+
|
|
399
|
+
## Google quotas
|
|
400
|
+
|
|
401
|
+
Requests per launched unit:
|
|
402
|
+
|
|
403
|
+
| Case | Reads | Writes |
|
|
404
|
+
|---|---|---|
|
|
405
|
+
| Unit whose mapped cells don't change | 1 | 0 |
|
|
406
|
+
| Ordinary unit | 2 (snapshot + conflict check) | 1 `batchUpdate` |
|
|
407
|
+
| `verify_before_write: false` | 1 | 1 |
|
|
408
|
+
| Appending a missing target row | +1 full used-range read | (same write) |
|
|
409
|
+
| Mapping with `gid` instead of `worksheet` | +1 metadata read | — |
|
|
410
|
+
| `dry_run` | 1 | 0 |
|
|
411
|
+
|
|
412
|
+
`push` makes one read, one conflict read and one write for any number of
|
|
413
|
+
units. `validate` makes one read; **Test** makes one metadata read and one
|
|
414
|
+
Drive permission read. A batch write counts as one request, however many
|
|
415
|
+
cells it changes.
|
|
416
|
+
|
|
417
|
+
The Sheets API limits requests per minute both per Cloud project and per
|
|
418
|
+
user (here, the service account). The published defaults are 300 per
|
|
419
|
+
project and 60 per user, counted separately for reads and writes. Check
|
|
420
|
+
current values under APIs & Services → Google Sheets API → Quotas. About 30
|
|
421
|
+
units finishing in one minute under one service account can therefore reach
|
|
422
|
+
the per-user read limit. 429 responses are retried up to three times within
|
|
423
|
+
the 25-second budget, and then the sync for that unit fails and is recorded.
|
|
424
|
+
Use `alfrd gsheet push` later to fill in what was missed, or give busy
|
|
425
|
+
projects separate service accounts.
|
|
426
|
+
|
|
427
|
+
## Troubleshooting
|
|
428
|
+
|
|
429
|
+
Failures never change a step's result. They appear in `runner.log`, as a
|
|
430
|
+
`plugin.hook_failed` notification, in `alfrd gsheet status` and in the Studio panel.
|
|
431
|
+
|
|
432
|
+
| Message | Cause and fix |
|
|
433
|
+
|---|---|
|
|
434
|
+
| `Google denied access (403, Sheets API): share the sheet …` | Share the spreadsheet with the key's `client_email` as Editor. |
|
|
435
|
+
| `Google denied access (403): the Drive API is disabled in the key's Google Cloud project …` | Only Test's edit check uses Drive. Open the printed link and enable it (allow a few minutes), or untick **Check edit permission with the Drive API**. |
|
|
436
|
+
| `Google denied access (403): the Sheets API is disabled …` | Enable the Sheets API with the printed link. |
|
|
437
|
+
| `Google denied access (403): … quota exceeded` | Too many requests per minute; it clears by itself. |
|
|
438
|
+
| `Google Sheet not found (404) …` | Wrong spreadsheet id or URL, or the sheet isn't shared with the service account. |
|
|
439
|
+
| `The service account has read-only access …` | It is shared as Viewer or Commenter; change it to Editor. |
|
|
440
|
+
| `Google rejected the service-account credentials (401).` / `Service-account authentication failed …` | The key was deleted or disabled, or the account was removed. Create a new key and paste it again. |
|
|
441
|
+
| `Service-account credentials must be valid JSON.` / `… must contain a service-account client_email and private_key.` | Paste the whole downloaded JSON file, not an OAuth client or an API key. |
|
|
442
|
+
| `Set a default spreadsheet id or URL before running Test.` | Test needs the default spreadsheet, even when mappings name their own. |
|
|
443
|
+
| `Google Sheet unavailable after 3 attempts (HTTP 429)` | Quota exceeded; see [Google quotas](#google-quotas). Backfill later with `push`. |
|
|
444
|
+
| `Google Sheet request deadline exceeded.` / `… sync time budget exhausted` | The network is slow or Google responded slowly. Lower `request_timeout` so retries fit in the budget. A timed-out write may still have landed; check with `diff`. |
|
|
445
|
+
| `Google Sheet sync lock timed out` | Another unit or `push` held the sheet lock for the whole budget. It usually clears; otherwise look for a stuck `alfrd gsheet` process. |
|
|
446
|
+
| `unknown sheet column 'X'` / `N problems: unknown column …` | A header was renamed or deleted, or a `{step}` wildcard names a column that doesn't exist. Rename the header or fix the rule; then run `validate`. |
|
|
447
|
+
| `ambiguous sheet column …` | Two headers have the same name after trimming and ignoring case. Rename one, or use the column letter. |
|
|
448
|
+
| `duplicate sheet row identity …` | The key column repeats a target. Add `rows.code_column` or remove the duplicate row. |
|
|
449
|
+
| `unknown step 'x'` | The rule names a step that isn't in the project's workflow. |
|
|
450
|
+
| Cells listed under *conflicts* and not written | Someone edited the cell while the step ran (`conflict_policy: skip`). Rerun `push` once the edit is resolved, or choose `overwrite`. |
|
|
451
|
+
| Nothing is written | Check that `enabled: true` is set and the target is in the key column. With `missing_row: skip`, absent targets are ignored, and the `when` filters must match. A `Dry run` chip or setting writes nothing. |
|
|
452
|
+
| Panel says "No step has finished since sync was set up." | No hooked unit has finished yet. The legacy `alfrd runtime` worker path has no hooks; use `push`. |
|
|
453
|
+
|
|
454
|
+
## Studio panel and validation
|
|
455
|
+
|
|
456
|
+
After installing and restarting Studio, the panel appears automatically for any
|
|
457
|
+
project with an `alfrd.gsheet.yaml`, including disabled or invalid mappings. An
|
|
458
|
+
explicit `views.metadata` entry is optional; existing entries keep working and
|
|
459
|
+
are not duplicated:
|
|
460
|
+
|
|
461
|
+
```yaml
|
|
462
|
+
views:
|
|
463
|
+
metadata:
|
|
464
|
+
- panel: gsheet_sync
|
|
465
|
+
title: Google Sheet
|
|
466
|
+
scope: project
|
|
467
|
+
```
|
|
468
|
+
|
|
469
|
+
The panel reads only `alfrd.gsheet.yaml` and `.alfrd/gsheet/sync.jsonl`. Opening it
|
|
470
|
+
makes no Google requests. It shows the newest sync per step, ten initially, with
|
|
471
|
+
**Show all** to expand the table. Counts are unit totals when a unit covers several
|
|
472
|
+
steps. History scanning is limited to the latest 1 MiB; `alfrd gsheet status` can
|
|
473
|
+
inspect older records. Interrupted or malformed history lines are ignored.
|
|
474
|
+
|
|
475
|
+
Use **Google Sheet: validate mapping** in the command palette for an explicit live,
|
|
476
|
+
read-only check of the selected project. It shares validation with the CLI and
|
|
477
|
+
reports mapping problems or connection errors in a toast. The request uses the
|
|
478
|
+
host's authenticated, loopback and CSRF-protected project-check endpoint.
|
|
479
|
+
|
|
480
|
+
Because the panel reads no global settings, its dry-run chip reflects the most
|
|
481
|
+
recent recorded sync. A sheet id inherited from Settings becomes a link after the
|
|
482
|
+
first recorded sync. A mapping with a numeric `gid` links directly to that tab;
|
|
483
|
+
worksheet titles link to the spreadsheet because resolving a title to a gid would
|
|
484
|
+
require a Google metadata request. Validate after changing global settings. The
|
|
485
|
+
panel checks intrinsic mapping syntax; workflow step and live-column checks belong
|
|
486
|
+
to the explicit validation command.
|
|
487
|
+
|
|
488
|
+
## Studio
|
|
489
|
+
|
|
490
|
+
Overview shows a **Google Sheet ▾** menu next to **Export** (it appears after restarting
|
|
491
|
+
`alfrd serve` with the plugin enabled). Everything below works without a terminal
|
|
492
|
+
and never edits `alfrd.yaml`.
|
|
493
|
+
|
|
494
|
+
- **Connect & validate…** opens the connection dialog, with sheet name, worksheet, mapping facts and last sync. A sheet without a mapping shows **Not connected** and **Connect sheet…**.
|
|
495
|
+
A saved mapping shows **Connected**; this indicates attachment, not a live
|
|
496
|
+
connectivity check. Use **Validate** to check sheet access. An unreadable mapping
|
|
497
|
+
shows **Needs attention**, with its error and recovery actions.
|
|
498
|
+
- **Connect sheet** (or the palette command **Google Sheet: attach…**) opens a
|
|
499
|
+
three-step dialog: paste a sheet URL or ID (or use the Settings default), **Load**
|
|
500
|
+
it, pick the tab and header row, then the target-name column (guessed from
|
|
501
|
+
`TARGET_NAME`, `target`, `source`, `name`) and an optional project-code column.
|
|
502
|
+
The first key values and how many match the project's plan targets are shown before
|
|
503
|
+
you attach. A URL with `#gid=` preselects that tab. Without a service-account key
|
|
504
|
+
the connection dialog links to Settings → Plugins → Google Sheet instead.
|
|
505
|
+
- **Edit mapping** (or **Google Sheet: open mapping editor**) edits outbound rules
|
|
506
|
+
(step, column, field with usage/results key, format, when, template), inbound rules
|
|
507
|
+
and options. Rows can be added, duplicated, moved and deleted; **Add status columns
|
|
508
|
+
for all steps** adds the `status` rules `init` would add. Problems from live
|
|
509
|
+
validation (600 ms after a change) appear next to the control and in a summary;
|
|
510
|
+
**Save** stays disabled until they are fixed. Saving rewrites the file in a stable
|
|
511
|
+
order and removes comments, so the first save asks for confirmation. If the file
|
|
512
|
+
changed on disk since it was opened, Save offers **Reload** instead of overwriting.
|
|
513
|
+
- **Validate connection**, **Preview changes…** (cell, target, step, column, old → new; the first 500 are
|
|
514
|
+
listed) and the menu’s **Export to sheet…** (preview first, then "Write N cells to <tab>?"). With
|
|
515
|
+
Settings dry run on, backfill writes nothing and says so.
|
|
516
|
+
- The menu’s **Sync** focuses the **Update this sheet during runs** switch and **Detach…**: disable sync and keep the file, or delete
|
|
517
|
+
`alfrd.gsheet.yaml` (sync history stays).
|
|
518
|
+
|
|
519
|
+
The menu, connection dialog, attach and editor call protected project actions. `plugin_actions: false` in
|
|
520
|
+
`plugins.json` (or `alfrd serve --no-plugin-actions`) disables Attach, Save, the switch,
|
|
521
|
+
Backfill and Detach; Validate and Preview stay available. Mapping edits check a SHA-256
|
|
522
|
+
revision under `.alfrd/locks/alfrd.gsheet.yaml.lock` and replace the file atomically;
|
|
523
|
+
CLI init uses the same lock, so hand edits and the Studio never silently overwrite
|
|
524
|
+
each other.
|
|
525
|
+
|
|
526
|
+
The form uses labelled sections, associated helper text, visible keyboard focus
|
|
527
|
+
and a single-column layout on narrow screens. The mapping tables scroll inside
|
|
528
|
+
their own labelled regions. See [OVERVIEW-UX.md](OVERVIEW-UX.md) and [STUDIO-UX.md](STUDIO-UX.md) for the presentation
|
|
529
|
+
specification and the proposed file-matching and column-creation flows (those two
|
|
530
|
+
flows are not implemented yet).
|