tasksource 0.1.0__tar.gz → 0.1.1__tar.gz

{tasksource-0.1.0 → tasksource-0.1.1}/.github/workflows/tests.yml

@@ -16,6 +16,6 @@ jobs:
       - uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
-      - run: python -m pip install . pytest
+      - run: python -m pip install . pytest requests
       - run: python -c "import tasksource; tasksource.list_tasks(); tasksource.list_tasks(multilingual=True)"
       - run: python -m pytest -q tests

tasksource-0.1.1/AGENTS.md

@@ -0,0 +1,36 @@
+# What enters the catalog
+
+A source becomes a training task only if all three hold:
+- it has a genuine training split (test-only benchmarks go to `eval_only.py`);
+- it is a decision: classification, multiple choice or token labeling, not open-ended answers;
+- its labels are human, procedural (solver, verifier, construction) or otherwise demonstrably reliable; unverified model labels do not qualify.
+
+Valid but low-signal sources get `max_examples`; systematically unreliable ones go to `parked.py`.
+
+# Dataset sources
+
+Reading an existing dataset with a mapping or a little code is an annotation: declare it in the task catalog on the original Hugging Face dataset. Heavy preprocessing is a repackage. Renaming or dropping columns is not a reason to mirror.
+
+Use existing Hugging Face datasets directly when `datasets.load_dataset` provides usable data. Keep task mappings declarative and reuse the existing templates and shared loader.
+
+For sources that require archive extraction, joins, broken-loader repairs, expensive pixel transformations such as crop generation, or have no usable Hub upload, add a reproducible conversion under `scripts/repackage_dataset/` (the upload entry point remains `scripts/upload_repackaged.py`) and publish a data-only `tasksource/` mirror. Pin upstream revisions, preserve native splits and original image bytes, record provenance and license evidence, and point the task declaration at the mirror. For generated crops, also record the actual pixel rectangle, encoding/configuration, and original action/widget identity. Avoid putting download or filesystem setup into task preprocessing.
+
+Keep visual mirrors consistent: ordered `images` plus canonical `inputs`, `choices_list` (for MC), `labels`, and JSON-string `metadata`. When many QAs share images, group those canonical QA records in `qa` and store the images once. Keep rationales and provenance in metadata, separate from model inputs.
+
+Each repackaged dataset README must link to the original data distribution and the specific reproducible packaging script, with source revisions, hashes and conversion settings recorded in linked provenance. A link to the packaging directory alone is insufficient.
+
+# Dataset documentation
+
+Keep Hub root dataset cards focused on current usage, configs, schema and license scope. Put release notes in CHANGELOG.md or addition manifests, and config-specific details in that config’s README. Incremental publication scripts must preserve the root card body rather than append batch summaries.
+
+# Jev releases
+
+The text configs of `tasksource/tasksource-jev-typed-decisions` (`default`, `full`) are rebuilt by `scripts/build_jev_dataset.py`, which replaces all their shards. New sources enter through the catalog, not through one-off publication scripts.
+
+To add sources without a rebuild, declare and pin them in the catalog, add the changelog entry, then run `python -m scripts.append_jev --config full|vision --tasks TASK ... --push --release vYYYY.MM.DD.n` (see `docs/jev/README.md`). It appends training rows only; published shards stay byte-identical, and `default` changes at the next full rebuild.
+
+# Changelog
+
+Every change that affects published data (new or removed sources, label fixes, sampling, schema) gets an entry in the root [`DATA_CHANGELOG.md`](DATA_CHANGELOG.md), newest first: `## <dataset> <version>` and a few concise, user-facing bullets on what changed in the data, not how it was done. Code-only changes need no entry. Keep history out of dataset cards.
+
+`tasksource-jev-typed-decisions` versions are `vYYYY.MM.DD[.n]`: add the entry before publishing; the build publishes that dataset's entries as its Hub `CHANGELOG.md` and tags the revision.

tasksource-0.1.1/CITATION.cff

@@ -0,0 +1,28 @@
+cff-version: 1.2.0
+message: "If you use tasksource, its standardized task annotations, or datasets derived from them in published research, please cite the tasksource paper."
+title: "tasksource"
+type: software
+authors:
+  - family-names: "Sileo"
+    given-names: "Damien"
+repository-code: "https://github.com/sileod/tasksource"
+license: BSD-3-Clause
+preferred-citation:
+  type: conference-paper
+  title: "tasksource: A Large Collection of NLP tasks with a Structured Dataset Preprocessing Framework"
+  authors:
+    - family-names: "Sileo"
+      given-names: "Damien"
+  collection-title: "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)"
+  conference:
+    name: "LREC-COLING 2024"
+    city: "Torino"
+    country: "IT"
+  publisher:
+    name: "ELRA and ICCL"
+  year: 2024
+  month: 5
+  start: 15655
+  end: 15684
+  doi: "10.63317/2h4pkr685mkq"
+  url: "https://aclanthology.org/2024.lrec-main.1361"

tasksource-0.1.1/DATA_CHANGELOG.md

@@ -0,0 +1,317 @@
+# Data changelog
+
+Changes that affect published data, newest first: one entry per publish, `## <dataset> <version>`, with a few
+concise bullets on what changed in the data. `tasksource-jev-typed-decisions` releases are tagged on the Hub
+(`load_dataset("tasksource/tasksource-jev-typed-decisions", revision="v2026.10.13")`).
+
+## tasksource-jev-typed-decisions v2026.10.16.1
+
+- `vision`: add RefCOCO, RefCOCO+ and RefCOCOg object-box decisions, sharing the existing 20k training-row family budget. The source mirror contains 100k training QAs.
+- Native object boxes provide the choices; images, grounding geometry, source identities and original license evidence remain available.
+- Append training rows only, excluding images already in published evaluation. Existing training and evaluation shards remain unchanged.
+
+## tasksource/refcoco-grounding v2026.10.16
+
+- Add 100,000 training, 24,527 validation and 30,903 test referring-expression decisions from RefCOCO, RefCOCO+ and RefCOCOg.
+- Share unchanged COCO2014 images across the three views. Choices are native object boxes; no synthetic distractors.
+- Reserve held-out photographs across variants by both image ID and image hash. Preserve original split names in metadata; native test subsets share the canonical test split.
+- Record original per-image Flickr licenses and source revisions. Noncommercial and unresolved terms are retained explicitly.
+
+## tasksource-jev-typed-decisions v2026.10.16
+
+- `vision`: add `spatial-dise/mcq` with native spatial-reasoning choices and original composite images. Training excludes generator families reserved by validation, test or the separate Spatial-DISE benchmark.
+- Added 2,410 training decisions; `vision` now has 662,314 train rows. Existing shards and evaluation are unchanged.
+- Retain rotation, combination and hidden-face questions. Projection is omitted for missing viewing cues; folding for a sampled ambiguous answer set. Source terms are CC BY-NC-SA 4.0 with original third-party asset rights retained.
+
+## tasksource/spatial-dise v2026.10.16.1
+
+- Remove cube-folding questions after a sampled hard example had multiple visually compatible answers. The approved mirror retains 2,410 train, 793 validation and 1,065 test questions.
+
+## tasksource 2026-10-16
+
+- Add `spatial-dise/mcq` through a reproducible image-archive mirror: 2,410 train, 793 validation and 1,065 test questions. Native answer panels remain identifiable after option permutation; no fabricated choices.
+
+## tasksource-jev-typed-decisions v2026.10.15.5
+
+- `vision`: removed the grid-cell grounding rows (rico-widget/grid7: 15,180; spair71k/grid7: 30,000); the coordinate-option views of v2026.10.15.4 replace them. Other rows and shards are unchanged; vision train has 659,904 rows.
+
+## tasksource-jev-typed-decisions v2026.10.15.4
+
+- `vision`: coordinate grounding as multiple choice. RICO widgets and Mind2Web elements: pick the target's box among the screen's other element boxes, as 0–1000 `[x1, y1, x2, y2]`. SPair-71k: pick the corresponding point among the pair's other keypoints, as 0–1000 `(x, y)`.
+- `vision`: Mind2Web training actions (next action type, element choice, box choice); its test splits stay out.
+- 75,258 training rows added: rico-widget/box 30,000, spair71k/point 30,000, and 5,086 each for Mind2Web box, element and action.
+- Grid-cell and axis-bin grounding views no longer enter Jev builds; the rows already published leave at the next vision rebuild.
+
+## tasksource 2026-10-15.12
+
+- New tasks `procedural-choices/reasoning_core` (54 tasks) and `procedural-choices/reasoning_gym` (68 tasks): verified procedural answers among options their scorer rejects, without near-correct options.
+- New visual tasks `rico-widget/box`, `mind2web/box` and `spair71k/point`: native candidate boxes or keypoints as 0–1000 coordinate options.
+
+## tasksource-jev-typed-decisions v2026.10.15.3
+
+- Rebuild of `default` and `full`. Rows keep their published `id` and `example_id` when their source example is unchanged, also where the content changed.
+- Removed sources with unreliable or redundant labels: gen_debiased_nli, hlgd, EVALution, blog_authorship/job, app_reviews, pragmeval mrda and EmoBank arousal/valence, brainteasers/SP, cos_e/v1.0, crowdflower nuclear sentiment.
+- At most 1,000 training rows from robust_nli, pragmeval scales, logical-fallacy, path-naturalness, crowdflower political media, recast_verbcorner and ReSQ; 10,000 from KnowRef-60K.
+- Readable option labels (e.g. "answerable from the passage", "e2 causes e1", TREC answer types) and task questions for MultiRC, ethics/justice, ethics/virtue and dnd_style_intents.
+- `default` now also draws on the sources appended to `full` since the last rebuild.
+
+## zero-shot-label-nli 2026-10-15
+
+- Removed the parked sources listed above, and tasks whose labels are bare yes/no or true/false answers to a task question.
+- Readable label hypotheses ("This example is answerable from the passage.") for the relabeled tasks; rare civil_comments attributes removed.
+
+## tasksource 2026-10-15.11
+
+- civil_comments threat, sexual_explicit, obscene, identity_attack and severe_toxicity parked: few positives, flagged by keyword. toxicity, insult and the rater-share tasks remain.
+- Task questions for MultiRC, ethics/justice, ethics/virtue and dnd_style_intents; "Receive quest" label spelling.
+
+## tasksource 2026-10-15.10
+
+- KnowRef-60K capped at 10,000 training rows: some sentences stay ambiguous despite 4/5 annotator agreement.
+
+## tasksource 2026-10-15.9
+
+- Parked cos_e/v1.0 (its questions are CommonsenseQA's, with fewer choices) and crowdflower/sentiment_nuclear_power (171 mostly neutral tweets, some mislabeled).
+- Capped at 1,000 training rows: crowdflower political-media audience, bias and message (subjective labels), recast_verbcorner (templated) and ReSQ (questions about objects the story omits).
+
+## tasksource 2026-10-15.8
+
+- Self-describing labels for yes/no tasks with a fixed question, e.g. "plausible answer" (MC-TACO), "answerable from the passage" (SQuAD v2, QuAC, CUAD), "same entity" (PreCo), "causal relation" (causal-news), "correct replacement" (ReCoRD), and wiki_qa, wikiqa, avicenna, ambient, I2D2, winowhy, conceptrules_v2, regset, PARARULE-Plus, missing-item-prediction.
+- zero-shot-label-nli leaves out tasks whose labels are bare yes/no or true/false answers to a task question.
+
+## tasksource-jev-typed-decisions v2026.10.15.2
+
+- Append CBT named-entity/common-noun cloze, ReCAM, high-agreement KnowRef-60K and native text-table TabMWP decisions to `full`, with all native choices retained.
+- Exclude malformed choices, error/low-agreement labels and shared source holdouts, including ReCAM's unlabeled-test articles. Add source/license provenance and keep solutions out of model inputs.
+- Preserve existing training shards and evaluation data. `default` gains these sources at the next canonical rebuild.
+
+## tasksource 2026-10-15.7
+
+- Readable labels where codes or bare words hid the judgment: TREC answer types (with a question), SemEval-2010 relations ("e2 causes e1"), FewRel ("the relation holds"), ESCI query relevance, defeasible-nli ("strengthens the hypothesis", with premise and hypothesis marked), com2sense ("makes sense"), and negative/positive, not spam, not hate for IMDb, Rotten Tomatoes, SMS spam and hate_speech18.
+
+## tasksource 2026-10-15.6
+
+- Parked after checking flagged examples against the raw sources: hlgd (loose story groupings), EVALution lexical relations (wrong source relations), blog_authorship/job (industry not inferable) and app_reviews (arbitrary ratings on short reviews).
+- logical-fallacy capped at 1,000 training rows (noisy labels).
+- brainteasers/SP removed: the mirror's sentence-puzzle files were a copy of the word puzzles.
+- path-naturalness-prediction capped at 1,000 training rows. sen-making/1 and path-naturalness-prediction carry their question as the input instead of an empty one.
+
+## tasksource 2026-10-15.5
+
+- Add CBT named-entity/common-noun cloze, ReCAM abstract-word cloze, KnowRef-60K pronoun resolution and native TabMWP text-table multiple choice.
+- Preserve all native choices, including CBT's ten and ReCAM's five. Filter CBT's malformed candidates and KnowRef's error/low-agreement labels (require 4/5); retain annotation provenance separately from prompts.
+- Reserve shared passages/tables for source holdouts, including ReCAM's original unlabeled test articles. Keep available native partitions and do not fabricate new splits.
+
+## tabmwp 2026-10-15
+
+- Publish native multiple-choice questions with complete textual tables, original answer options and solutions kept in metadata. Retain 5,317 train, 1,830 validation and 1,992 test questions after table-group protection.
+- Preserve original CC BY-NC-SA 4.0 dataset terms, native source IDs, file hashes and reproducible conversion/exclusion records.
+
+## tasksource-jev-typed-decisions v2026.10.15.1
+
+- `vision`: +150,000 training rows (479,826 → 629,826) from procedural visual puzzles, 30,000 each: TRON, TRACE, Mentis Oculi paper folding, SPHINX and VisGym. Existing rows and evaluation splits are unchanged.
+
+## tasksource-jev-typed-decisions v2026.10.15
+
+- Append FaithDial original-response grounding, RAGTruth evidence-grounding judgments and ComVE commonsense statement choices to `full`, using human supervision and checked native labels.
+- Preserve existing training shards and evaluation data. `default` includes these sources at the next canonical rebuild.
+- Retain annotation provenance and source license evidence; exclude generic/uncooperative FaithDial cases, low-quality RAGTruth outputs and invalid or conflicting ComVE pairs.
+
+## tasksource 2026-10-15.5
+
+- New `max_examples` task annotation caps training rows for valid but low-signal sources; every consumer (Jev `full`, recasts) respects it.
+- Capped at 1,000 training rows: the robust_nli stress tests and the median-split pragmeval scales (EmoBank dominance, Squinky, persuasiveness).
+- Parked: the six gen_debiased_nli variants (generated hypotheses, model-filtered labels), pragmeval/mrda (context-free meeting dialogue acts) and the EmoBank arousal/valence median splits.
+
+## tasksource 2026-10-15.4
+
+- Add FaithDial original-response grounding and RAGTruth human-span-derived grounding decisions. Keep annotation provenance separate from inputs, reserve native dialogue/source holdouts, and omit ambiguous or low-quality cases.
+- Add ComVE native two-statement commonsense choices; invert the source's nonsense index to select the more sensible statement. Exclude duplicate choices, repeated pairs and conflicting judgments.
+- AAAC and GLUCOSE remain deferred: their proposed MC conversions need verified negatives rather than arbitrary distractors.
+
+## comve 2026-10-15
+
+- Publish official human-written subtask-A statement pairs as a data-only mirror, with native splits, original answer indices, per-file hashes and exclusion reasons.
+- Preserve CC BY-SA 4.0 licensing and link the original release and reproducible packaging script.
+
+## tasksource 2026-10-15.3
+
+- Add `sbic/offensiveness` and `sbic/intent` from the audited aggregate, with 11,692 and 8,647 accepted training judgments respectively. Preserve native uncertainty categories and labeling details in metadata.
+- Keep SBIC_Disagreement unchanged. Group-target supervision is withheld; its original annotations remain available in metadata.
+- The source mirror is published; these text tasks enter Jev `full`/`default` only through the canonical rebuild, not a partial append.
+
+## sbic 2026-10-15
+
+- Aggregate SBIC crowd judgments once per post and worker, preserving native categorical ratings and all labeling provenance in metadata.
+- Require at least three valid workers and 75% agreement per label. Keep only posts with a reliable offensiveness or intent judgment; missing answers never count as negatives.
+- Preserve native splits, assigning duplicate posts to their highest-priority source holdout. Exclude posts conflicting with the existing SBIC_Disagreement partitions. Retain original group-target annotations in metadata, with their correct mapping documented; the group-target task is withheld after a questionable unanimous negative in the source audit.
+
+## tasksource 2026-10-15.2
+
+- Add `trip/cloze` and `trip/order`: two native story choices with human physical-plausibility labels, using the official Hugging Face Parquet conversion directly.
+- Preserve the original 799/322/351 and 2,330/393/408 train/validation/test partitions. Story groups do not overlap across splits, including between views.
+- Record the original license as unspecified. These tasks enter the next canonical Jev text build; they have not been published to `full`/`default` yet.
+
+## tasksource 2026-10-15
+
+- Add `quality`, `clear/readability`, `esconv/strategy`, `prosocial-dialog/safety` and `gap` to the English catalog. Preserve full reading passages, native options, raw reading-ease scores and source partitions.
+- Keep earlier conversation context without future turns or annotation rationales; shared ProsocialDialog opening contexts favor evaluation splits. Preserve native marked pronoun spans and record human/model input provenance and original licenses.
+- MedNLI remains access-restricted; LegalBench is excluded by request. These catalog additions are queued for the next canonical text build, not yet published to the Jev `full`/`default` configs.
+
+## esconv 2026-10-15
+
+- Preserve the official train/validation/test dialogue partitions and every native supporter-strategy label in a data-only mirror.
+- Keep consecutive replies with different strategies separate. Classify the current reply using prior context only; retain conversation identity and source provenance. Original CC BY-NC 4.0 terms apply.
+
+## tasksource-jev-typed-decisions v2026.10.13.7
+
+- `vision`: +170,008 training rows (309,818 → 479,826) from larger source mirrors, up to 30,000 rows per family: BAPPS, DocLayNet, LVIS and COCO panoptic regions (+29,000 each), SPair-71k (+25,008), Super-CLEVR (+29,000 across six views). Existing rows and evaluation splits are unchanged.
+
+## visgym-mcq 2026-10-14.2
+
+- Consolidate VisGym into one balanced release: 100,000 training QAs, exactly 25,000 each for mazes, equation repair, stick rotation and block legality; 25 validation QAs per family.
+- Deterministically shuffle the combined dataset, retaining verified native choices, original PNGs, family/geometry metadata and disjoint image groups. Replace the separate family registrations with `visgym/mcq`.
+- Delete the superseded standalone equation, rotation and block Hub repositories after consolidation.
+
+## zero-shot-label-nli 2026-10-14
+
+- Rebuilt from the current catalog: 305 classification tasks (from 152), 1.84M training rows; `default` and `triplet` configs as before.
+- Per task, the majority label is at most 3× the others, and tasks with only true/false/yes/no labels are capped at 1,000 training rows.
+- Left out: tasks with NLI or annotator-agreement labels, or any bare-number label (e.g. HelpSteer scales). Adds `license` and `license_use` columns.
+
+## visgym-equation-mcq 2026-10-14
+
+- Add a reviewed fresh procedural pilot for one-match equation repair. Every offered equation is arithmetically true, and exactly one is reachable with one match transfer.
+- Preserve generation seeds, geometry, Apache-2.0 source notices and native/independent candidate checks. Target a separate 100,000-QA verified training build; exclude cross-split image copies.
+
+## visgym-rotation-mcq 2026-10-14
+
+- Add a reviewed fresh procedural pilot for stick-orientation questions with nearby 15/30-degree alternatives, checked with the native orientation reward.
+- Preserve generation seeds, geometry, Apache-2.0 source notices and native/independent candidate checks. Target a separate 100,000-QA verified training build; exclude cross-split image copies.
+
+## visgym-block-mcq 2026-10-14
+
+- Add a reviewed fresh procedural pilot for sliding-block move legality. Distractors collide with another block; only the current board is displayed.
+- Preserve generation seeds, geometry, Apache-2.0 source notices and native/independent candidate checks. Target a separate 100,000-QA verified training build; exclude cross-split image copies.
+
+## sphinx-mcq 2026-10-14
+
+- Add fresh static questions from four reviewed SPHINX families: pie ordering, transformation identification, transformation result selection and geometric counting; target 100,000 distinct training QAs.
+- Keep native same-scene distractors and image panel identities; use nearby count alternatives. Preserve source metadata, generation seeds, MIT code and CC BY 4.0 icon notices.
+
+## visgym-mcq 2026-10-14
+
+- Add 100,000 distinct fresh VisGym 2D maze training QAs and 100 validation QAs at three sizes. Select the unique shortest-path next move from four cardinal directions.
+- Preserve maze geometry and seed provenance; exclude tied or trivial states and cross-split image copies. Scope excludes photographic and simulator assets.
+
+## tron-mcq 2026-10-14.2
+
+- Keep all 100,000 training QAs, original PNGs, choices and gold indices. Clarify numeric questions requesting two decimal places as choosing the closest offered value; some source geometry figures display rounded measurements while their gold uses hidden precision.
+- Preserve original questions in metadata and record the repair against the pinned 100k release.
+
+## trace-mcq 2026-10-14.2
+
+- Retain the native release and add fresh procedural training instances to reach 100,000 verified QAs; native validation remains separate.
+- Use independent generation seeds and same-task native training answers as distractors. Exclude validation image copies by decoded pixels as well as repeated image/question/gold combinations.
+- Remove three additional standalone JSON-output directives from the MC questions while preserving source content, including chart labels named “Json”. Record fresh-generation seeds, rendering versions and resume checkpoints.
+
+## mentis-oculi-paper-fold 2026-10-14
+
+- Expand fresh procedural paper-fold training to 100,000 distinct, geometrically verified QAs, retaining native five-panel images and choices.
+- Preserve separate held-out seed namespaces, remove image overlap and repeated questions, and record all chunk seeds and exclusions. Original benchmark rows are not ingested.
+
+## trace-mcq 2026-10-14
+
+- Expand selection to the complete eligible native training release (64,000 source rows) and native validation partition (2,000 source rows), without duplication to reach an artificial 100,000-row target.
+- Keep original image bytes, same-task verifier-checked distractors, source partitions, repaired prompt contracts and asset attribution. Retained counts and exclusions are recorded in provenance.
+
+## tron-mcq 2026-10-14
+
+- Expand the five reviewed procedural environments to 100,000 distinct, verifier-checked training QAs.
+- Keep native PNGs and research-use terms. Remove repeated image/question/gold combinations and held-out image copies; record resumable generation chunks and actual seeds.
+
+## tasksource-jev-typed-decisions v2026.10.13.6
+
+- `vision`: +191,169 training rows (118,649 → 309,818), up to 30,000 rows per source family: NLVR2, Visual7W, TallyQA, MapQA, A-OKVQA, RICO widgets, CLEVR, SPair-71k, GameQA, Valen game, AI2D, ScienceQA, TQA. Families without more unseen rows are unchanged; existing rows and evaluation splits are unchanged.
+
+## mentis-oculi-paper-fold 2026-10-13
+
+- Add a fresh procedural pilot: 20 training and 5 validation paper-fold puzzles with five native image panels. Published benchmark examples are not ingested.
+- Recheck final unfolded option geometry for exactly one matching answer. Record generation seeds, coordinates, source revision and Apache-2.0 terms.
+
+## trace-mcq 2026-10-13
+
+- Add 5,000 training and 100 validation visual multiple-choice questions from TRACE's native release, preserving original image bytes and partitions.
+- Distractors come from training answers for the same task and answer type, checked with the original scorer. Remove JSON output demonstrations, clarify image-label choices, and restore the missing toggle-grid transition rule from source code.
+- Exclude all native validation image copies from training. Preserve CC BY 4.0 attribution and upstream asset notices.
+
+## tron-mcq 2026-10-13
+
+- Add a bounded static visual MC release from five seeded TRON environments: aquarium, angle bisector, bar-chart aggregate, before/after and overlapping-shape count.
+- Keep native text choices where available; otherwise use verifier-rejected answers from other generations of the same environment. Reject ambiguous choices and unsupported answer formats.
+- Preserve generated PNGs, record seeds and verifier results, and exclude held-out image copies from training. Mark TRON's original research-use terms explicitly.
+
+## tasksource-jev-typed-decisions v2026.10.13.5
+
+- Add native single-answer MemeQA and four English MET-Meme classification views. Training is capped at 20,000 rows per source family; existing training and evaluation shards are preserved.
+- Image-only memes remain usable with an empty text state. Answer metadata stays out of inputs. MemeQA retains its native meme splits; MET-Meme retains its Hub partitions with held-out image copies and conflicting duplicate-image labels removed. MemeQA image terms are marked research-use; MET-Meme's license is explicitly unspecified.
+
+## memeqa 2026-10-13
+
+- New reproducible image mirror: 3,000 train, 1,244 validation and 1,194 test native single-answer MCQs, with original image bytes and meme-level partitions.
+- Excludes native multiselect questions, empty/invalid choices and duplicate options; no Tasksource-generated answers or distractors. Annotation provenance records human verification of automatically constructed source distractors; image research-use terms remain separate from the repository's MIT license.
+
+## t2i-safety 2026-10-13
+
+- New generated-image toxicity mirror: 30,781 train and 366 test judgments, with native safe/unsafe labels and an explicit six-category policy. Original image bytes and split membership are preserved; image groups are shuffled with seed 0.
+- Generation prompts stay out of inputs. Demographic prediction, privacy-only/mixed labels and unreviewed real images are outside this view. Registered for the next canonical vision build with a 20,000-training-row family cap.
+
+## uk-election 2026-10-13
+
+- New native target-sentiment mirror: 9,911 train and 2,675 test judgments, preserving the official tweet partitions and human-added targets. “Does not apply” and one invalid span are excluded.
+- Training view is parked after a coarse review found unsupported topic sentiment. Annotation license is CC BY 4.0; tweet-content rights remain separate.
+
+## gameqa 2026-10-13
+
+- Image groups shuffled within each native split using seed 0; all questions, labels and encoded image bytes are preserved. Existing Jev rows are unchanged.
+
+## tasksource-jev-typed-decisions v2026.10.13.4
+
+- `source_index` (`split:row` of the original dataset row) on every row, to trace label errors back to their source.
+- Stable row order across rebuilds: shuffled by `example_id`, so row-level annotations stay valid.
+- Removed after manual review of random rows: `dair-ai/emotion`, CrowdFlower `text_emotion` and IndicGLUE ACTSA (many wrong labels). `app_reviews` drops reviews under four words.
+- DiscoSense states include the discourse connective; PropSegmEnt question fixed; PROST removed (test-only).
+- New: CEBaB, FABSA, TAB, NewsMTSC, SentiHood target/aspect sentiment and MAMS-ATSA.
+- Restores 57 sources missing from v2026.10.13 (MAUD, multidogo, WikiQA, OLID, TabFact and others), now built with the rest instead of appended.
+
+## tasksource-jev-typed-decisions v2026.10.13.3
+
+- `vision`: GameQA and Valen Game grow to 20,000 training rows each, retaining the original 1,000-row pilots (106,085 vision training rows total). All new Valen rows are English; its preserved bilingual pilot leaves 19,493 English and 507 Chinese rows.
+- New rows are shuffled together with seed 0; original shards and evaluation splits are unchanged. GameQA question identities now include the native record to avoid collisions.
+
+## tasksource-jev-typed-decisions v2026.10.13.1
+
+- `vision`: game decisions, 1,000 training rows each from [GameQA](https://huggingface.co/datasets/OpenMOSS-Team/GameQA-140K) multiple-choice questions and [Valen](https://huggingface.co/datasets/Valen-Team/Valen-Eval-Game) optimal-action judgments (68,085 vision training rows). Existing rows and evaluation splits are unchanged.
+
+## tasksource-jev-typed-decisions v2026.10.13
+
+- 18 procedural sources, including `record_eligibility`; procedural rows are label-balanced (at most 60% per answer, yes/no at most a third of questions).
+- Label balance: per source and question, the majority label is at most three times the rarest; single-label questions are dropped.
+- `default` mix weighs sources by option count, so binary decisions are 15% of rows.
+- Known issue: 58 sources from earlier appends are missing (MAUD, multidogo, WikiQA, OLID, TabFact and others).
+
+## procedural-typed-decisions 2026-10-13
+
+- New task `record_eligibility`; `partial_observation_calibration` recalibrated so levels go from easy (L0) to hard (L4).
+- Answers balanced: at most 60% per answer within a question, and yes/no questions at most a third of each task.
+
+
+## tasksource-jev-typed-decisions: earlier releases
+
+Untagged, before 2026-10-13.
+
+- Text sources added: WebInstruct (MC and binary); web-page decisions (Mind2Web, WebLINX, WebSRC); SEntFiN, WANDS and SciRepEval Search; legal, biomedical and structured text (MAUD, ACORD, CUAD, Evidence Inference, TabFact, WikiSQL, MultiDoGO and others); dialogue and evidence (QuAC, MultiWOZ, SLURP, QASPER and others); stance, argument and discourse (FNC-1, UKPConvArg, DiscoGeM, PubMed RCT); linguistic annotations (OLID, iSarcasm, MAGPIE, SemCor, STREUSLE and others). Spider aggregation was later withdrawn.
+- 2026-10-09: CLadder removed (evaluation-only). Missing-item, WikiHow order and QuaRel questions fixed; QuaRel uses its answer texts as options.
+- 2026-10-09, `vision`: answer-style suffixes removed from 18,353 Cauldron questions; component-level license review in `metadata.licenses.license_review`. FigureQA rows are flagged: its terms prohibit redistribution.
+- 2026-10-08: `vision` config, a multimodal pilot of about 66k training decisions from 30 sources, with `license` and `license_use` columns.
+- `filtered-full`: a filtered, training-only snapshot of an earlier release; its exclusions are recorded as reusable evidence in `quality/filtered-full-v1/`.

tasksource-0.1.1/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2022-2026, Damien Sileo
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

tasksource-0.1.1/PKG-INFO

@@ -0,0 +1,172 @@
+Metadata-Version: 2.4
+Name: tasksource
+Version: 0.1.1
+Summary: Standardized decision tasks (classification, multiple choice, token labeling) for multi-task training and evaluation
+Home-page: https://github.com/sileod/tasksource/
+License: BSD-3-Clause
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Intended Audience :: Developers
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: LICENSE-CC-BY-4.0
+Requires-Dist: dotwiz
+Requires-Dist: funcy
+Requires-Dist: datasets
+Requires-Dist: Pillow
+Requires-Dist: exrex
+Requires-Dist: ftfy
+Requires-Dist: magicattr
+Requires-Dist: pandas
+Requires-Dist: numpy
+Requires-Dist: scipy
+Requires-Dist: sorcery
+Requires-Dist: langcodes[data]
+Dynamic: license-file
+
+## <img src="https://cdn-avatars.huggingface.co/v1/production/uploads/5fc0bcb41160c47d1d43856b/j06-U5e2Tifi2xOnTudqS.jpeg" alt="" height="32" align="absmiddle"> tasksource: 1,100+ decision tasks, standardized for interchangeable use
+
+tasksource is a collection of **decision tasks**: classification, multiple choice and token labeling,
+with negative examples or classes, standardized so that they can be used interchangeably for
+multi-task training, evaluation and reward modeling. Labels come from human annotation or from
+procedures that verify them, and the focus is on training data: evaluation benchmarks are kept separate.
+
+Hugging Face Datasets is an excellent library, but datasets lack a common format and often need
+preprocessing before they can be used together. Each tasksource annotation maps a dataset to one of three
+templates with canonical fields: `Classification`, `MultipleChoice` or `TokenClassification`. An annotation
+is a short, readable function from a dataset to its standardized form.
+
+- 570 English, 550 multilingual and 45 vision tasks: browse the [English](catalog_english.md),
+  [multilingual](catalog_multilingual.md) and [vision](catalog_vision.md) catalogs.
+- Soft labels (annotator votes and rating distributions), license filters, and per-task questions that say what to decide.
+- Recasts to typed decisions ([tasksource-jev-typed-decisions](#tasksource-jev-typed-decisions)),
+  [instructions](#tasksource-instruct) and [label NLI](#tasksource-label-nli).
+
+### Usage
+
+```bash
+pip install tasksource
+```
+
+```python
+from tasksource import list_tasks, load_task
+
+df = list_tasks(multilingual=False) # takes some time
+for id in df[df.task_type=="MultipleChoice"].id:
+    dataset = load_task(id) # all yielded datasets can be used interchangeably
+```
+
+The [example catalog](docs/examples.md) shows one real example per task: input, question, options and
+gold answer. Evaluation benchmarks are kept out of the training catalogs; load them with
+`from tasksource import eval_only`. Feel free to request a new task.
+
+Datasets are downloaded to `$HF_DATASETS_CACHE`, like any Hugging Face dataset; large multi-task runs
+need more than 100GB.
+
+#### Vision
+
+`list_tasks(vision=True)` lists the visual tasks and `load_task(id, vision=True)` loads them, with images.
+
+#### Questions
+
+Inputs are kept raw. When they do not say what to predict, a task carries a `question`
+("Is this search query a well-formed question?"), exposed as `dataset.question`;
+`load_task(id, prompted=True)` appends it to the inputs.
+
+#### Soft labels
+
+Some tasks are distributions rather than single labels: annotator votes, rater shares, survey counts.
+`load_task(id, soft=True)` loads them as probabilities over the options, and `list_tasks(soft=True)`
+lists them. By default, `load_task(id)` gives the majority label on rows with clear agreement.
+
+#### Licenses
+
+`list_tasks(license_use="commercial")` keeps tasks whose sources allow commercial use (also
+`non-commercial`, `unspecified`, or a list), and `task_licenses()` gives each task's licenses and their
+sources. This is a best-effort filter, not legal advice.
+
+The tasksource code is under the [BSD 3-Clause license](LICENSE). Original documentation and the task
+annotations we write (mappings, questions, label names, recasts) are under [CC BY 4.0](LICENSE-CC-BY-4.0).
+Third-party datasets, including the sources that tasksource loads and our mirrors or repackages of
+them, keep their own licenses, which `task_licenses()` reports.
+
+### tasksource-jev-typed-decisions
+
+Tasks can be recast as typed decisions: a state and one or more questions of three kinds, `choice`
+(pick one option), `score` (an ordered scale) and `noul` (yes/no or a probability). This is the Jev / System One request format.
+
+[🤗 tasksource/tasksource-jev-typed-decisions](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions)
+is the published corpus:
+
+| config | content |
+|---|---|
+| `default` | 1M train rows, a mix steered by source quality, difficulty and option count |
+| `full` | 2.5M train rows from every source |
+| `vision` | decisions over images |
+
+It includes
+[🤗 tasksource/procedural-typed-decisions](https://huggingface.co/datasets/tasksource/procedural-typed-decisions):
+generated states (ledgers, schedules, graphs, policies, records) with exact labels at five difficulty levels.
+
+```python
+from tasksource import load_task, render_typed_decision
+
+dataset = load_task("glue/rte", recast="jev")
+request = render_typed_decision(dataset["train"][0], model="openjev")
+```
+
+Multiple-choice options are shuffled per row and over-represented labels are capped, so neither the gold
+position nor the most common label pays. Benchmarks such as MMLU, BIG-bench and BLiMP are excluded.
+See the [dataset card](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions) for the row format and the mixes.
+
+### Pretrained models
+
+Text encoder pretrained on tasksource reached state-of-the-art results: [🤗/deberta-v3-base-tasksource-nli](https://hf.co/sileod/deberta-v3-base-tasksource-nli)
+
+Tasksource pretraining is notably helpful for RLHF reward modeling or any kind of classification, including zero-shot. You can also find a large and a multilingual version.
+
+### tasksource-instruct
+
+Tasks recast as instructions (`recast="instruct"`), one of the richest instruction-tuning datasets:
+[🤗/tasksource-instruct](https://hf.co/datasets/tasksource/tasksource-instruct)
+
+### tasksource-label-nli
+
+Classification tasks recast as natural language inference, for entailment-based zero-shot classification:
+[🤗/zero-shot-label-nli](https://huggingface.co/datasets/tasksource/zero-shot-label-nli)
+
+### Write and use custom preprocessings
+
+```python
+from tasksource import MultipleChoice
+
+codah = MultipleChoice('question_propmt',choices_list='candidate_answers',
+    labels='correct_answer_idx',
+    dataset_name='codah', config_name='codah')
+
+winogrande = MultipleChoice('sentence',['option1','option2'],'answer',
+    dataset_name='winogrande',config_name='winogrande_xl',
+    splits=['train','validation',None]) # test labels are not usable
+
+tasks = [winogrande.load(), codah.load()]  # aligned datasets (same columns) can be used interchangeably
+```
+
+### Citation and contact
+
+For more details, refer to the [article](https://arxiv.org/abs/2301.05948):
+```bib
+@inproceedings{sileo-2024-tasksource,
+    title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
+    author = "Sileo, Damien",
+    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
+    month = may,
+    year = "2024",
+    address = "Torino, Italia",
+    publisher = "ELRA and ICCL",
+    url = "https://aclanthology.org/2024.lrec-main.1361",
+    doi = "10.63317/2h4pkr685mkq",
+    pages = "15655--15684",
+}
+```
+For help integrating tasksource into your experiments, please contact [damien.sileo@inria.fr](mailto:damien.sileo@inria.fr).