ellf-cli 19.0.4__tar.gz → 19.0.5__tar.gz

{ellf_cli-19.0.4/ellf_cli.egg-info → ellf_cli-19.0.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ellf-cli
-Version: 19.0.4
+Version: 19.0.5
 Summary: Ellf Command Line Interface
 Home-page: https://prodi.gy
 Author: ExplosionAI GmbH

{ellf_cli-19.0.4 → ellf_cli-19.0.5}/ellf_cli/ellf.json

@@ -1,7 +1,7 @@
 {
   "prog": "ellf",
   "help": "Ellf Command Line Interface.",
-  "version": "19.0.4",
+  "version": "19.0.5",
   "extra_key": "_extra",
   "commands": {
     "actions": {

{ellf_cli-19.0.4 → ellf_cli-19.0.5}/ellf_cli/ellf_skills/skills/ellf-annotate.assistant/SKILL.md

@@ -27,9 +27,9 @@ If the workflow requires a new custom recipe or custom interface, use
 methodology or schema work is needed, use `/ellf-project`.
 
 **Never narrate routing or argument inference to the user.** The user does
-not need to know that you're delegating to `/ellf-ops`, what `auto_start`
-value you inferred from their verb, or which cli-name you chose for which
-field. Say only what the user can act on: the outcome, the next step, the
+not need to know that you're delegating to `/ellf-ops`, whether you
+inferred from their verb that they want the task started, or which cli-name
+you chose for which field. Say only what the user can act on: the outcome, the next step, the
 link. The audit summary is useful when something failed and the user needs
 to fix it; otherwise skip it and let the create tool's artifact (card,
 form link, or prerequisite link) do the talking.
@@ -118,17 +118,20 @@ mcp__pam__task_create(
   recipe_id=<from recipe_list>,
   args=<resolved scalar args>,
   name=<optional; PAM auto-names from recipe + timestamp when omitted>,
-  auto_start=<true if the user said run/start/launch, false for create/save/set up>,
 )
 ```
 
+Creating never starts the task. If the user said run/start/launch and the
+create returned `status: "created"`, call `mcp__pam__task_start(id=<object.id>)`
+next; for create/save/set up, stop after creation.
+
 If an annotation agent was also planned, call `mcp__pam__agent_create(...)`
 for the agent. After the agent is created and started, attach it to the
 task with `mcp__pam__task_assign_bot(task_id=..., agent_id=...)`.
 
 The create tool's internal validation routes every case to a useful
 user-facing artifact:
-- confirmation card (three buttons: `Create and start` / `Create only` / `Cancel`) on clean validation
+- Allow/Deny confirmation card on clean validation (creates the task; it does not start it)
 - form-handoff link on missing scalars or complex JSON args
 - missing-prerequisite link on a referenced asset / dataset / secret that doesn't exist on the cluster
 
@@ -143,8 +146,8 @@ X, Y, Z", any pre-emptive validation reasoning).
 Once the user confirms the create card (or the form-handoff link is followed
 through to completion), help verify and run any follow-ups:
 
-- if the task wasn't auto-started (user clicked `Create only`): use `/ellf-ops`
-  start workflow when the user is ready
+- if the user didn't ask to run it yet: use the `/ellf-ops` start workflow when
+  they're ready
 - check cluster status with `mcp__cluster__job_status(id="<id>")` to confirm the
   task is running or healthy
 - provide the task link and tell the user where to open it in the app

{ellf_cli-19.0.4 → ellf_cli-19.0.5}/ellf_cli/ellf_skills/skills/ellf-annotate.assistant/references/annotation_audit.md

@@ -1,7 +1,7 @@
 # Annotation Audit Reference
 
 Quick-reference for auditing an annotation plan before launching a recipe.
-This covers the checks ellf-annotate runs in Step 6. For deep strategic
+This covers the checks ellf-annotate runs before launch. For deep strategic
 questions (team structure, agile workflow, full schema redesign), escalate to
 ellf-project.
 
@@ -112,12 +112,11 @@ These catch cases where a different recipe would be significantly more efficient
 
 | Situation | Suggestion |
 |---|---|
-| User picks `ner.manual` but has a trained model | Suggest `ner.correct` — faster (correct vs annotate from scratch) |
-| User picks `ner.manual` but has existing patterns | Use patterns for pre-annotation, then `ner.correct` to review |
-| User picks `textcat.manual` but has a trained classifier | Suggest `textcat.correct` |
-| User picks `spans.manual` but has a trained spans model | Suggest `spans.correct` |
-| Large dataset, no model assistance | Suggest active learning recipe (`.teach` variants) |
-| Complex UI needs (custom blocks, multiple views) | Delegate to ellf-prodigy for custom recipe |
+| `ner` or `spans` with a blank model, but a trained pipeline exists | Pick the trained model in the recipe's **Model** field: the task shows its predictions to correct, faster than annotating from scratch |
+| `ner` or `spans` without patterns, but patterns exist | Add them as a `patterns` asset in the **Patterns** field to pre-highlight matches |
+| `textcat` without a model, but a trained classifier exists | Pick the classifier in the **Model** field to pre-select its predictions |
+| Large dataset, no model assistance | Consider an annotation agent (`gemini_agent`) to pre-annotate, with humans correcting; see `builtin_ellf_annotation_recipes.md` |
+| Complex UI needs (custom blocks, multiple views) | Hand the custom recipe to the coding agent with `/ellf-handoff` |
 
 ---
 

{ellf_cli-19.0.4 → ellf_cli-19.0.5}/ellf_cli/ellf_skills/skills/ellf-annotate.assistant/references/builtin_ellf_annotation_recipes.md

@@ -39,9 +39,27 @@ workflows.
 | `audio` | Audio annotation |
 | `video` | Video annotation |
 | `curate` | Data curation / exploration |
-| `review` | Review and correction of prior annotations |
+| `review` | Review and correction of prior annotations (structured datasets only) |
 | `sent` | Sentence segmentation |
 
+PDF task recipes (the `recipes_pdf` package; confirm the cluster lists them):
+
+| Recipe | Use for |
+|---|---|
+| `pdf_image` | Bounding boxes and segments on PDF pages rendered as images |
+| `pdf_spans` | Text spans in PDFs, from a `layout` asset made by the `pdf_layout_fetch` action |
+| `pdf_ocr_correct` | OCR the boxes from a `pdf_image` dataset and correct the extracted text |
+
+`pdf_image` and `pdf_ocr_correct` take a `pdf` asset (`pdf_ocr_correct` also the
+`pdf_image` dataset); `pdf_spans` takes the `layout` asset that `pdf_layout_fetch`
+makes from one. The `pdf_ingest` action renders a PDF annotation-ready
+(it can also run on upload), and `pdf_layout_fetch` extracts its text and layout into the `layout`
+asset `pdf_spans` needs.
+
+`review` only reads datasets in structured storage. Datasets with examples from before
+it must first go through the "Migrate dataset to structured" action
+(`migrate_to_structured`). Its labels default to the ones found in the annotations.
+
 ## Label schemes
 
 Only `ner`, `spans` and `textcat` take a label scheme. No other built-in recipe accepts
@@ -61,7 +79,7 @@ for an existing one before writing a new one. Format and design advice:
 
 ## Annotation instructions
 
-Built-in task recipes take annotation instructions in their annotation settings
+Built-in task recipes except `review` and `terms` take annotation instructions in their annotation settings
 (the **Instructions** field): an asset of kind `instructions` holding one Markdown
 document. HTML isn't accepted. Annotators open it in the annotation app from the help
 icon or with the H key.
@@ -108,15 +126,20 @@ them when the user asks for Hugging Face models or already has a suitable fine-t
 model.
 - `hf_model_agent` runs a model already fine-tuned for the task (from the Hub or an
   `hf_model` asset) on the cluster. It only serves `ner_manual` / `spans_manual`, binary
-  `ner` / `spans`, `choice` and binary `classification` tasks. Every task label (or
-  choice option) must map to a model label via `label_map`. If one doesn't, or the
-  interface isn't supported (`blocks`, `html`, relations, image, audio), the agent stops
-  on its first task with an error naming what's missing, rather than answering.
+  `ner` / `spans`, `choice` and binary `classification` tasks. Model labels with the
+  same name as a task label map automatically; map the others with `label_map` (model
+  label → task label). It stops on its first task with an error naming what's missing
+  when a manual spans or choice task has a label the model can't produce, when the
+  interface isn't supported, or when the model type doesn't fit the task (a
+  token-classification model on a choice task, say). Binary tasks whose label the model
+  doesn't have are answered "ignore" instead.
 - `hf_llm_agent` works like `gemini_agent` but uses a chat model from Hugging Face
   Inference Providers, and needs an `hf_token` secret. A bad token, missing credits or an
   unusable model stops it at startup with the reason.
 
-The LLM agents (`gemini_agent`, `hf_llm_agent`) include the task's annotation
+The LLM agents (`gemini_agent`, `hf_llm_agent`) only annotate text: tasks without
+text, and interfaces no prompt covers (relations, dependencies, `blocks`, image, audio),
+are answered "ignore" and left to people. They include the task's annotation
 instructions in every prompt by default. Their **Agent annotation settings** add:
 - **Agent instructions**: an agent-only `instructions` asset, combined with the task's.
   Use it for guidance only the model needs, such as mistakes it keeps making.
@@ -143,4 +166,4 @@ definitions.
   annotations rather than creating annotations from scratch.
 - If the user needs a custom interface, multi-view workflow, unusual routing,
   or annotation logic the built-ins cannot express cleanly, delegate to
-  `ellf-prodigy`.
+  `/ellf-handoff`, which hands the custom recipe to the coding agent.

{ellf_cli-19.0.4 → ellf_cli-19.0.5}/ellf_cli/ellf_skills/skills/ellf-annotate.assistant/references/label_schemes.md

@@ -9,10 +9,10 @@ into something a task can use, or when writing annotation guidelines for labels
 that already exist.
 
 **Only some recipes take a scheme.** A scheme is a recipe argument, and the only
-built-in recipes that have it are `ner`, `spans` and `textcat` (`--label-scheme`, or
-**Asset** on the **Labels** field). A custom recipe can take one too, if it implements
+built-in recipes that have it are `ner`, `spans` and `textcat` (**Asset** on the
+**Labels** field). A custom recipe can take one too, if it implements
 it with `props.label_scheme` (hand a new one off to the coding agent with `/ellf-handoff`). No other built-in recipe accepts a
-`labels` asset, so never suggest a scheme or `--label-scheme` for one of them. For
+`labels` asset, so never suggest a scheme for one of them. For
 those, write the label definitions into the annotation guidelines (an `instructions`
 asset) instead, or build a custom recipe if the definitions need to reach annotators
 and agents label by label.
@@ -93,23 +93,23 @@ Show it to the user and settle the labels before publishing.
 
 ## Publishing and using a scheme
 
-- **Publish:** `ellf publish data scheme.json "<cluster_path>" --name "<name>" --kind labels`,
-  or upload the file on the cluster's Assets page with the kind `labels`. Assets can be
-  previewed from the Assets page and from the asset picker in recipe forms.
-- **Reuse:** check for an existing `labels` asset (`ellf assets list`, or the Assets
-  page) before writing a new one. Use one scheme per label set, not one per task.
+- **Publish:** the user uploads the file on the cluster's Assets page with the kind
+  `labels`. Assets can be previewed from the Assets page and from the asset picker in
+  recipe forms.
+- **Reuse:** ask the user to check the Assets page for an existing `labels` asset
+  before writing a new one. Use one scheme per label set, not one per task.
 - **Tasks:** only the built-in `ner`, `spans` and `textcat` recipes (and custom recipes
-  that implement it) take a scheme. They accept either a typed list of labels
-  (`--label A,B`) or a scheme (`--label-scheme <asset>`). In the web form, the
-  **Labels** field switches between **List** and **Asset**. Prefer a scheme whenever
+  that implement it) take a scheme. In the web form, their **Labels** field switches
+  between **List** (typed labels) and **Asset** (a scheme). Prefer a scheme whenever
   the labels need explaining, are reused across tasks, or agents will annotate.
 - **Annotators** see a label's description when they hover the label in the annotation
   app (and on choice options for text classification).
 - **Agents:** a task with a scheme passes its label definitions on to every LLM agent assigned to it
   (`gemini_agent`, `hf_llm_agent`), and they're included in each prompt: description,
   `do_not_mark` and the examples that fit the task type. An agent can have its own
-  scheme on top (`--labels.scheme`), whose definitions replace the task's for the
-  labels it defines, or ignore the task's definitions (`--labels.ignore-task-labels`).
+  scheme on top (its **Label scheme** setting), whose definitions replace the task's
+  for the labels it defines, or ignore the task's definitions (**Ignore label
+  definitions from task**).
   The agent always annotates the task's labels; labels only its scheme has are ignored.
   `hf_model_agent` doesn't use label definitions.
 

{ellf_cli-19.0.4 → ellf_cli-19.0.5}/ellf_cli/ellf_skills/skills/ellf-annotate.coding/SKILL.md

@@ -162,13 +162,15 @@ A launch is not complete until you verify that the annotation process is usable.
 ### Ellf cluster
 
 ```bash
-ellf tasks info "<name>" --json
-ellf tasks logs "<name>"
+ellf tasks logs "<name>" --show-errors --json   # state + structured errors
+ellf tasks info "<name>" --json                 # url + stats
 ```
 
-Confirm the task exists and is running. If an agent was requested:
+Confirm the task exists and its state is `running` or `ready` (`info --json` has no
+state field). If an agent was requested:
 
 ```bash
+ellf agents logs "<agent_name>" --show-errors --json
 ellf agents info "<agent_name>" --json
 ```
 

{ellf_cli-19.0.4 → ellf_cli-19.0.5}/ellf_cli/ellf_skills/skills/ellf-annotate.coding/references/annotation_audit.md

@@ -1,7 +1,7 @@
 # Annotation Audit Reference
 
 Quick-reference for auditing an annotation plan before launching a recipe.
-This covers the checks ellf-annotate runs in Step 6. For deep strategic
+This covers the checks ellf-annotate runs before launch. For deep strategic
 questions (team structure, agile workflow, full schema redesign), escalate to
 ellf-project.
 

{ellf_cli-19.0.4 → ellf_cli-19.0.5}/ellf_cli/ellf_skills/skills/ellf-annotate.coding/references/builtin_ellf_annotation_recipes.md

@@ -39,9 +39,27 @@ workflows.
 | `audio` | Audio annotation |
 | `video` | Video annotation |
 | `curate` | Data curation / exploration |
-| `review` | Review and correction of prior annotations |
+| `review` | Review and correction of prior annotations (structured datasets only) |
 | `sent` | Sentence segmentation |
 
+PDF task recipes (the `recipes_pdf` package; confirm the cluster lists them):
+
+| Recipe | Use for |
+|---|---|
+| `pdf_image` | Bounding boxes and segments on PDF pages rendered as images |
+| `pdf_spans` | Text spans in PDFs, from a `layout` asset made by the `pdf_layout_fetch` action |
+| `pdf_ocr_correct` | OCR the boxes from a `pdf_image` dataset and correct the extracted text |
+
+`pdf_image` and `pdf_ocr_correct` take a `pdf` asset (`pdf_ocr_correct` also the
+`pdf_image` dataset); `pdf_spans` takes the `layout` asset that `pdf_layout_fetch`
+makes from one. The `pdf_ingest` action renders a PDF annotation-ready
+(`ellf publish data <file.pdf> --kind pdf --render` runs it on upload), and `pdf_layout_fetch` extracts its text and layout into the `layout`
+asset `pdf_spans` needs.
+
+`review` only reads datasets in structured storage. Datasets with examples from before
+it must first go through the "Migrate dataset to structured" action
+(`migrate_to_structured`). Its labels default to the ones found in the annotations.
+
 ## Label schemes
 
 Only `ner`, `spans` and `textcat` take a label scheme. No other built-in recipe accepts
@@ -61,7 +79,7 @@ for an existing one first with `ellf assets list`. Format and design advice:
 
 ## Annotation instructions
 
-Built-in task recipes take annotation instructions in their annotation settings
+Built-in task recipes except `review` and `terms` take annotation instructions in their annotation settings
 (`--settings.instructions`): an asset of kind `instructions` holding one Markdown
 document. HTML isn't accepted. Annotators open it in the annotation app from the help
 icon or with the H key. Publish it with
@@ -109,15 +127,20 @@ them when the user asks for Hugging Face models or already has a suitable fine-t
 model.
 - `hf_model_agent` runs a model already fine-tuned for the task (from the Hub or an
   `hf_model` asset) on the cluster. It only serves `ner_manual` / `spans_manual`, binary
-  `ner` / `spans`, `choice` and binary `classification` tasks. Every task label (or
-  choice option) must map to a model label via `label_map`. If one doesn't, or the
-  interface isn't supported (`blocks`, `html`, relations, image, audio), the agent stops
-  on its first task with an error naming what's missing, rather than answering.
+  `ner` / `spans`, `choice` and binary `classification` tasks. Model labels with the
+  same name as a task label map automatically; map the others with `label_map` (model
+  label → task label). It stops on its first task with an error naming what's missing
+  when a manual spans or choice task has a label the model can't produce, when the
+  interface isn't supported, or when the model type doesn't fit the task (a
+  token-classification model on a choice task, say). Binary tasks whose label the model
+  doesn't have are answered "ignore" instead.
 - `hf_llm_agent` works like `gemini_agent` but uses a chat model from Hugging Face
   Inference Providers, and needs an `hf_token` secret. A bad token, missing credits or an
   unusable model stops it at startup with the reason.
 
-The LLM agents (`gemini_agent`, `hf_llm_agent`) include the task's annotation
+The LLM agents (`gemini_agent`, `hf_llm_agent`) only annotate text: tasks without
+text, and interfaces no prompt covers (relations, dependencies, `blocks`, image, audio),
+are answered "ignore" and left to people. They include the task's annotation
 instructions in every prompt by default. Their agent settings add:
 - `--settings.instructions`: an agent-only `instructions` asset, combined with the
   task's. Use it for guidance only the model needs, such as mistakes it keeps making.

{ellf_cli-19.0.4 → ellf_cli-19.0.5}/ellf_cli/ellf_skills/skills/ellf-annotate.coding/references/builtin_prodigy_recipes.md

@@ -16,6 +16,7 @@ built-ins or an existing custom recipe are not enough.
 | `ner.correct` | Correct model predictions |
 | `ner.teach` | Active-learning binary review |
 | `ner.llm.correct` | Correct LLM-assisted predictions |
+| `ner.silver-to-gold` | Turn binary (accept/reject) annotations into complete gold ones |
 
 ## Span Categorization
 
@@ -23,6 +24,7 @@ built-ins or an existing custom recipe are not enough.
 |---|---|
 | `spans.manual` | Manual overlapping span annotation |
 | `spans.correct` | Correct span predictions |
+| `spans.llm.correct` | Correct LLM-assisted span predictions |
 
 ## Text Classification
 
@@ -60,6 +62,30 @@ built-ins or an existing custom recipe are not enough.
 |---|---|
 | `coref.manual` | Manual coreference annotation |
 
+## Image and audio
+
+| Recipe | Use for |
+|---|---|
+| `image.manual` | Bounding boxes, polygons and freehand shapes on images |
+| `audio.manual` | Label regions in audio or video |
+| `audio.transcribe` | Transcribe audio or video |
+
+## Terminology
+
+| Recipe | Use for |
+|---|---|
+| `terms.teach` | Build a terminology list from seed terms with word vectors |
+| `terms.to-patterns` | Turn a terminology dataset into match patterns |
+
+## Pre-annotating and comparing
+
+| Recipe | Use for |
+|---|---|
+| `ner.model-annotate` / `spans.model-annotate` / `textcat.model-annotate` | Save a model's predictions to a dataset, e.g. to review or compare |
+| `ner.llm.fetch` / `spans.llm.fetch` / `textcat.llm.fetch` | Fetch LLM predictions for a whole corpus up front, then correct them |
+| `compare` | Compare two sets of outputs side by side (A/B) |
+| `ner.eval-ab` | A/B evaluation of two NER models |
+
 ## Relations and review
 
 | Recipe | Use for |

{ellf_cli-19.0.4/ellf_cli/ellf_skills/skills/ellf-project.coding → ellf_cli-19.0.5/ellf_cli/ellf_skills/skills/ellf-annotate.coding}/references/label_schemes.md

@@ -64,8 +64,10 @@ no plain list of labels.
     exactly the span's `text`.
 
 Unknown keys are an error (`"desc"` instead of `"description"`), and so is anything
-else that doesn't fit the format. The file is checked when a task or agent loads it,
-and every problem is reported at once, so fix them all in one go.
+else that doesn't fit the format; a top-level `"$schema"` string is allowed. The file
+is checked when it's published with `ellf publish data --kind labels` and again when a
+task or agent loads it, and every problem is reported at once, so fix them all in one
+go.
 
 ## Designing a scheme
 
@@ -99,25 +101,34 @@ user has annotated data for the labels, or can't settle the definitions up front
 
 - **From an existing dataset.** Export it with
   `ellf datasets export "<dataset>" -o annotations.jsonl` and derive the scheme from
-  it, as described below.
+  it, as described below. Export refuses datasets that still hold examples from
+  before structured storage; run the "Migrate dataset to structured" action
+  (`migrate_to_structured`) on them first.
 - **Pilot first, then write the scheme.** If there's no data yet and the labels
   aren't settled, queue up a small batch with a plain list of labels (`--label A,B`)
   and flagging enabled (`"show_flag": true` in the Prodigy config), and ask the user
   to go through 50–100 examples, flagging the ones they're unsure about. Then build
-  the scheme from that dataset, and restart the task, or create the next one, with
-  `--label-scheme`, so annotators and agents see the definitions from then on. The
+  the scheme from that dataset, publish it, and create the next task with
+  `--label-scheme`, so annotators and agents see the definitions from then on (a
+  task's recipe args can't be changed after creation). The
   flagged examples are usually what `do_not_mark` and the boundary examples need.
 
+The export is structured JSONL, one row per answer: the text is in `input.text`,
+the annotation in `user_anns` (`user_anns.spans` for spans, `user_anns.label` for
+text classification, `user_anns.accept` for choice; a key missing there comes from
+`server_anns`, the suggestion as served), the answer in `answer`, and the flag in
+`extras.flagged`. Rows don't record who annotated them.
+
 Derive the scheme from the exported JSONL:
 - **Labels and order:** the labels used in the data, most frequent first.
 - **`description`:** what the accepted spans or texts of each label have in common,
   written as a rule, not a summary of the data.
 - **`do_not_mark`:** the near misses. In binary (accept/reject) datasets, rejected
   suggestions are near misses. In manual datasets, look for the same kind of phrase
-  annotated inconsistently, examples with `"flagged": true`, and texts that annotators
-  answered differently (compare `_session_id`s on the same `_input_hash`).
+  annotated inconsistently, rows with `extras.flagged` set, and texts that got
+  different answers or annotations (rows sharing the same `input_hash`).
 - **`examples`:** one clear positive and one boundary case per label, copied from
-  accepted answers. Prodigy spans already have `"start"` and `"end"`: keep them when
+  accepted answers. The spans in `user_anns.spans` already have `"start"` and `"end"`: keep them when
   the span text appears more than once in the example, and drop `token_start`,
   `token_end` and the other keys. Skip `"ignore"`d answers, and rejected answers in
   manual datasets, which mean the whole example was bad.

{ellf_cli-19.0.4 → ellf_cli-19.0.5}/ellf_cli/ellf_skills/skills/ellf-handoff/SKILL.md

@@ -44,7 +44,6 @@ bound at session time.
 | Field | Value |
 |---|---|
 | **Request ID** | `<id from tool result>` |
-| **Project** | <project name> |
 | **Status** | planned |
 | **Description** | <description> |
 
@@ -53,3 +52,10 @@ bound at session time.
 
 Briefly explain what the task covers — enough for the user to understand,
 not a full implementation plan.
+
+## Later: checking status
+
+When the user asks how a handoff went, call `todo_list()` (scoped to this
+conversation). With more than one result, let the user pick via `AskUserQuestion`,
+using each todo's description as the label. Then `todo_read(id=...)` returns its
+status and the resolution the coding agent wrote.

{ellf_cli-19.0.4 → ellf_cli-19.0.5}/ellf_cli/ellf_skills/skills/ellf-monitor.assistant/SKILL.md

@@ -55,7 +55,8 @@ If there are multiple plausible jobs, use `AskUserQuestion` to let the user choo
 
 Read:
 - `${CLAUDE_SKILL_DIR}/references/training_monitoring.md`
-- `${CLAUDE_SKILL_DIR}/../ellf-train/references/training_troubleshooting.md`
+- `${CLAUDE_SKILL_DIR}/../ellf-train/references/diagnostics.md` when an alert needs a root cause
+
 Use cluster calls such as:
 ```text
 mcp__cluster__job_status(id="<id>")
@@ -79,9 +80,9 @@ Use cluster calls for job status and per-dataset counts; use PAM for dataset
 discovery:
 ```text
 mcp__cluster__job_status(id="<id>")
-mcp__pam__dataset_list(cluster_id="<id>")               # discover datasets (org-wide, registered with PAM)
-mcp__cluster__dataset_example_count(name="<name>")      # total examples in the dataset (per-cluster, via broker)
-mcp__cluster__dataset_session_counts(name="<name>")     # per-annotator breakdown (keys are session_ids; null = bulk import)
+mcp__pam__dataset_list()                                # discover datasets (org-wide), then pick the one you need by name
+mcp__cluster__dataset_example_count(name="<name>")      # {active, annotated, used, legacy} (per-cluster, via broker)
+mcp__cluster__dataset_session_counts(name="<name>")     # {counts: {<session>: {annotated, active, used}}, removed: [...]}
 ```
 
 `dataset_list` is a PAM read (org-wide registry), while the count tools talk to
@@ -95,7 +96,8 @@ Read when needed:
 Report:
 - task state
 - whether the task appears reachable and healthy
-- annotation count (from `dataset_example_count`)
+- annotation count (from `dataset_example_count`: `active` for the dataset's size,
+  `annotated` for the work done)
 - per-annotator activity (from `dataset_session_counts`) when the user asks
   who annotated what or about active annotators
 - dataset growth
@@ -201,4 +203,4 @@ When you finish, state:
 |------|---------------|--------------|
 | `${CLAUDE_SKILL_DIR}/references/training_monitoring.md` | Training log interpretation, alert classification, metric extraction | Training actions |
 | `${CLAUDE_SKILL_DIR}/references/annotation_metrics.md` | Annotation progress signals, dataset growth, annotator activity | Annotation tasks |
-| `${CLAUDE_SKILL_DIR}/../ellf-train.assistant/references/diagnostics.md` | Six problem classes with detection signals and fix guidance | Diagnosing training issues |
+| `${CLAUDE_SKILL_DIR}/../ellf-train/references/diagnostics.md` | Six problem classes with detection signals and fix guidance | Diagnosing training issues |

ellf_cli-19.0.5/ellf_cli/ellf_skills/skills/ellf-monitor.assistant/references/annotation_metrics.md

@@ -0,0 +1,34 @@
+# Annotation Metrics
+
+Use this reference to interpret annotation activity and quality signals.
+
+## What to watch
+
+- Annotation count — from `mcp__cluster__dataset_example_count`: `active` is what the
+  dataset holds now (its size), `annotated` is all the work done, including versions
+  later edited over or removed. `legacy` > 0 means examples from before structured
+  storage that the counts leave out until the dataset is migrated.
+- Dataset growth over time — compare counts across checks
+- Who annotated what — from `mcp__cluster__dataset_session_counts`. Each key in
+  `counts` is a session dataset: `{dataset}-{task}-{user}` for an annotator's work on a
+  task, plus `-{session}` keys for single sittings. Each has `annotated`, `active` and
+  `used` (active and not ignored). The ids are user ids for people and agent ids for
+  agents (resolve them with `users_list` / `agent_list`). An annotator's figures and their sittings' overlap,
+  so count annotators from the `{dataset}-{task}-{user}` keys only and never add the
+  two kinds up. `removed` lists sittings removed from the dataset. Examples written
+  directly (e.g. via `db-in`) are in no session.
+- Whether a running task is producing new examples — combine job status with
+  example count over time
+
+## Warning signals
+
+- Very fast growth can indicate careless annotation
+- Very slow or flat growth can indicate task reachability or assignment problems
+- Frequent agent errors with no dataset growth usually mean the agent is failing to annotate
+- Repeated pauses or restarts may indicate an unstable launch or worker issue
+
+## Interpretation rules
+
+- Do not infer annotation quality from counts alone
+- Use counts and status to describe activity, not to prove correctness
+- If progress is suspicious, recommend operational checks first, then methodology review if needed

{ellf_cli-19.0.4 → ellf_cli-19.0.5}/ellf_cli/ellf_skills/skills/ellf-monitor.coding/SKILL.md

@@ -91,10 +91,16 @@ Use it when available because it keeps raw logs out of context and produces stru
 
 If needed, supplement with:
 ```bash
-ellf actions info "<name>" --json
+ellf actions logs "<name>" --show-errors --json   # state + structured errors
 ellf actions logs "<name>"
 ```
 
+`info --json` has no state field; take state from `logs --show-errors --json`
+(`pending`, `running`, `ready`, `succeeded`, `failed`, `stopped`, `cancelled`,
+`not_found`, or `unknown` when the cluster can't be reached) or from the `state`
+column of `ellf actions list --json`. For a job that was created with `--no-start` and
+never started, `logs` fails; use `list --json` instead.
+
 For local runs, inspect:
 - `job_status.json`
 - `training_log.jsonl`
@@ -111,10 +117,20 @@ Report:
 
 Use:
 ```bash
-ellf tasks info "<name>" --json
-ellf tasks logs "<name>"
+ellf tasks logs "<name>" --show-errors --json   # state + structured errors
+ellf tasks info "<name>" --json                 # url + stats (annotations, annotators, sessions)
+ellf tasks logs "<name>" [--query "<text>"]     # --query filters lines (needs Loki on the cluster)
+ellf sessions list "<name>" --json              # per session: annotator, annotated, active, status
 ```
 
+`sessions list` answers "who annotated how much": `annotated` is the work done,
+`active` how much of it the dataset still holds, and `status` shows excluded
+sessions. Excluding or re-activating one is an operation: route it to `/ellf-ops`.
+
+Don't use `ellf tasks questions` to peek at a live task: it reserves that batch for the
+CLI's session, so annotators won't get those questions until they're answered or
+stolen by another session.
+
 To check the linked dataset and confirm the output target:
 ```bash
 ellf datasets info "<dataset_name>" --json
@@ -133,7 +149,7 @@ Report:
 
 Use:
 ```bash
-ellf agents info "<name>" --json
+ellf agents logs "<name>" --show-errors --json   # state + structured errors
 ellf agents logs "<name>"
 ```
 
@@ -154,11 +170,21 @@ If a Hugging Face agent stopped soon after starting, its log's last error says w
   provider is left to automatic routing: store an HF token, or name the provider the
   key belongs to.
 
+## Orphaned workloads
+
+If the cluster seems busy or a job won't go away, list what's actually running:
+
+```bash
+ellf jobs list --unmatched --json   # workloads with no matching task/action/agent/service in Ellf
+```
+
+Stopping one (`ellf jobs stop <job_id>`) is an operation: route it to `/ellf-ops`.
+
 ## Generic actions
 
 Use:
 ```bash
-ellf actions info "<name>" --json
+ellf actions logs "<name>" --show-errors --json   # state + structured errors
 ellf actions logs "<name>"
 ```
 
@@ -247,5 +273,5 @@ When you finish, state:
 |------|---------------|--------------|
 | `${CLAUDE_SKILL_DIR}/references/training_monitoring.md` | Training log interpretation, alert classification, metric extraction | Training actions |
 | `${CLAUDE_SKILL_DIR}/references/annotation_metrics.md` | Annotation progress signals, dataset growth, annotator activity | Annotation tasks |
-| `${CLAUDE_SKILL_DIR}/../ellf-train.coding/references/diagnostics.md` | Six problem classes with detection signals and fix guidance | Diagnosing training issues |
-| `${CLAUDE_SKILL_DIR}/../ellf-train.coding/references/training_troubleshooting.md` | Error taxonomy with concrete fixes | Concrete setup or execution errors |
+| `${CLAUDE_SKILL_DIR}/../ellf-train/references/diagnostics.md` | Six problem classes with detection signals and fix guidance | Diagnosing training issues |
+| `${CLAUDE_SKILL_DIR}/../ellf-train/references/training_troubleshooting.md` | Error taxonomy with concrete fixes | Concrete setup or execution errors |

{ellf_cli-19.0.4 → ellf_cli-19.0.5}/ellf_cli/ellf_skills/skills/ellf-monitor.coding/references/training_monitoring.md

@@ -4,7 +4,7 @@ How to monitor training jobs, interpret alerts, and diagnose problems.
 
 ## The check script
 
-The ellf-train skill includes `scripts/check_training.py` — it reads local
+This skill includes `scripts/check_training.py` — it reads local
 logs or cluster status and produces a structured JSON summary.
 
 ```bash
@@ -18,7 +18,7 @@ python ${CLAUDE_SKILL_DIR}/scripts/check_training.py --mode cluster --job-name n
 Output shape:
 ```json
 {
-  "state": "running|completed|failed",
+  "state": "running|completed|failed|stopped|cancelled|pending|ready|not_found|unknown",
   "step": 800,
   "train_loss": 0.52,
   "dev_scores": {"ents_f": 0.79, "ents_p": 0.82, "ents_r": 0.76},
@@ -37,6 +37,12 @@ Output shape:
 | `plateau` | 8+ evals without score improvement | Present analysis | Yes |
 | `overfitting` | Dev score declining while train loss improves | Show best checkpoint info | Yes |
 | `completed` | Training finished normally | Report completion and results | No |
+| `stopped` / `cancelled` | Cluster job was stopped or cancelled before finishing | Report it; ask whether it was intended | Yes |
+
+`nan_loss`, `loss_spike`, `plateau` and `overfitting` come from the local
+`training_log.jsonl` (ellf_logger). Cluster `train` has no such log, so cluster runs only
+get the state alerts (`failed`, `stopped`, `cancelled`, `completed`); the step and best
+score come from the last 200 lines of the job log.
 
 ## Diagnostics