brainscore-language 2.1.0__tar.gz

brainscore_language-2.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 language-brainscore
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

brainscore_language-2.1.0/PKG-INFO

@@ -0,0 +1,87 @@
+Metadata-Version: 2.4
+Name: brainscore_language
+Version: 2.1.0
+License: MIT License
+        
+        Copyright (c) 2021 language-brainscore
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: tqdm
+Requires-Dist: numpy>=1.21
+Requires-Dist: brainscore-core
+Requires-Dist: fire
+Requires-Dist: scikit-learn
+Requires-Dist: torch>=1.9.1
+Requires-Dist: transformers>=4.11.3
+Requires-Dist: gensim
+Requires-Dist: joblib
+Requires-Dist: requests
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-check<=1.3.0; extra == "test"
+Dynamic: license-file
+
+[![Build Status](https://app.travis-ci.com/brain-score/language.svg?token=vqt7d2yhhpLGwHsiTZvT&branch=main)](https://app.travis-ci.com/brain-score/language)
+[![Documentation Status](https://readthedocs.org/projects/brain-score-language/badge/?version=latest)](https://brain-score-language.readthedocs.io/en/latest/?badge=latest)
+[![Website Status](https://img.shields.io/website.svg?down_color=red&down_message=offline&up_message=online&url=http%3A%2F%2Fwww.brain-score.org/language)](http://www.brain-score.org/language/)
+[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg)](code_of_conduct.md) 
+
+Brain-Score Language is a platform to evaluate computational models of language on their match to behavioral and neural
+measurements in the domain of language processing. The intent of Brain-Score is to adopt many (ideally all) the
+experimental benchmarks in the field for the purpose of model testing, falsification, and comparison. To that end,
+Brain-Score operationalizes experimental data into quantitative benchmarks that any model candidate following
+the `BrainModel` interface can be scored on.
+
+See the [Documentation](https://brain-score-language.readthedocs.io) for more details.
+
+Brain-Score is made by and for the community. To contribute,
+please [send in a pull request](https://github.com/brain-score/language/pulls).
+
+## Alpha status
+This repository is under active development, and should be considered to be in an alpha status.
+Code is likely to change substantially, without backwards compatibility.
+Please let us know of any problems by [creating an issue](https://github.com/brain-score/language/issues/new).
+
+## Setup
+
+```bash
+python -m pip install -e "." # change "." to ".[test]" to include testing dependencies
+```
+
+Please note that plugins (models, benchmarks, data, and metrics) may require additional dependencies beyond those installed here. Default behavior will install these dependencies directly into the runtime environment. 
+
+To avoid conflicts, we encourage either using an environment manager or setting the `BS_INSTALL_DEPENDENCIES` environment variable to either `no` or `newenv` (e.g. `export BS_INSTALL_DEPENDENCIES=newenv`). 
+
+`no` will leave all dependency installation up to the user, whereas `newenv` will create a new `conda` environment for the duration of the run. If you would like to use the `newenv` option and do not already have `conda` installed, you can follow the installation instructions [here](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html).
+
+## Usage
+
+Score an existing model on an existing benchmark:
+
+```bash
+python brainscore_language score --model_identifier='distilgpt2' --benchmark_identifier='Futrell2018-pearsonr'
+```
+
+## License
+
+MIT license

brainscore_language-2.1.0/README.md

@@ -0,0 +1,44 @@
+[![Build Status](https://app.travis-ci.com/brain-score/language.svg?token=vqt7d2yhhpLGwHsiTZvT&branch=main)](https://app.travis-ci.com/brain-score/language)
+[![Documentation Status](https://readthedocs.org/projects/brain-score-language/badge/?version=latest)](https://brain-score-language.readthedocs.io/en/latest/?badge=latest)
+[![Website Status](https://img.shields.io/website.svg?down_color=red&down_message=offline&up_message=online&url=http%3A%2F%2Fwww.brain-score.org/language)](http://www.brain-score.org/language/)
+[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg)](code_of_conduct.md) 
+
+Brain-Score Language is a platform to evaluate computational models of language on their match to behavioral and neural
+measurements in the domain of language processing. The intent of Brain-Score is to adopt many (ideally all) the
+experimental benchmarks in the field for the purpose of model testing, falsification, and comparison. To that end,
+Brain-Score operationalizes experimental data into quantitative benchmarks that any model candidate following
+the `BrainModel` interface can be scored on.
+
+See the [Documentation](https://brain-score-language.readthedocs.io) for more details.
+
+Brain-Score is made by and for the community. To contribute,
+please [send in a pull request](https://github.com/brain-score/language/pulls).
+
+## Alpha status
+This repository is under active development, and should be considered to be in an alpha status.
+Code is likely to change substantially, without backwards compatibility.
+Please let us know of any problems by [creating an issue](https://github.com/brain-score/language/issues/new).
+
+## Setup
+
+```bash
+python -m pip install -e "." # change "." to ".[test]" to include testing dependencies
+```
+
+Please note that plugins (models, benchmarks, data, and metrics) may require additional dependencies beyond those installed here. Default behavior will install these dependencies directly into the runtime environment. 
+
+To avoid conflicts, we encourage either using an environment manager or setting the `BS_INSTALL_DEPENDENCIES` environment variable to either `no` or `newenv` (e.g. `export BS_INSTALL_DEPENDENCIES=newenv`). 
+
+`no` will leave all dependency installation up to the user, whereas `newenv` will create a new `conda` environment for the duration of the run. If you would like to use the `newenv` option and do not already have `conda` installed, you can follow the installation instructions [here](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html).
+
+## Usage
+
+Score an existing model on an existing benchmark:
+
+```bash
+python brainscore_language score --model_identifier='distilgpt2' --benchmark_identifier='Futrell2018-pearsonr'
+```
+
+## License
+
+MIT license

brainscore_language-2.1.0/brainscore_language/__init__.py

@@ -0,0 +1,80 @@
+from typing import Dict, Any, Union, Callable
+
+from brainio.assemblies import DataAssembly
+from brainscore_core.benchmarks import Benchmark
+from brainscore_core.metrics import Score, Metric
+from brainscore_core.plugin_management.conda_score import wrap_score
+from brainscore_core.plugin_management.import_plugin import import_plugin
+from brainscore_language.artificial_subject import ArtificialSubject
+
+data_registry: Dict[str, Callable[[], Union[DataAssembly, Any]]] = {}
+""" Pool of available data """
+
+metric_registry: Dict[str, Callable[[], Metric]] = {}
+""" Pool of available metrics """
+
+benchmark_registry: Dict[str, Callable[[], Benchmark]] = {}
+""" Pool of available benchmarks """
+
+model_registry: Dict[str, Callable[[], ArtificialSubject]] = {}
+""" Pool of available models """
+
+
+def load_dataset(identifier: str) -> Union[DataAssembly, Any]:
+    import_plugin('brainscore_language', 'data', identifier)
+
+    return data_registry[identifier]()
+
+
+def load_metric(identifier: str, *args, **kwargs) -> Metric:
+    import_plugin('brainscore_language', 'metrics', identifier)
+
+    return metric_registry[identifier](*args, **kwargs)
+
+
+def load_benchmark(identifier: str) -> Benchmark:
+    import_plugin('brainscore_language', 'benchmarks', identifier)
+
+    return benchmark_registry[identifier]()
+
+
+def load_model(identifier: str) -> ArtificialSubject:
+    import_plugin('brainscore_language', 'models', identifier)
+
+    model = model_registry[identifier]()
+    model.identifier = identifier
+
+    return model
+
+def _run_score(model_identifier: str, benchmark_identifier: str) -> Score:
+    """
+    Score the model referenced by the `model_identifier` on the benchmark referenced by the `benchmark_identifier`.
+    """
+    model: ArtificialSubject = load_model(model_identifier)
+    benchmark: Benchmark = load_benchmark(benchmark_identifier)
+    score: Score = benchmark(model)
+    score.attrs['model_identifier'] = model_identifier
+    score.attrs['benchmark_identifier'] = benchmark_identifier
+    return score
+
+
+def score(model_identifier: str, benchmark_identifier: str, conda_active: bool=False) -> Score:
+    """
+    Score the model referenced by the `model_identifier` on the benchmark referenced by the `benchmark_identifier`.
+    The model needs to implement the :class:`~brainscore_language.artificial_subject.ArtificialSubject` interface
+    so that the benchmark can interact with it.
+    The benchmark will be looked up from the :data:`~brainscore_language.benchmarks` and evaluates the model
+    (looked up from :data:`~brainscore_language.models`) on how brain-like it is under that benchmark's
+    experimental paradigm, primate measurements, comparison metric, and ceiling.
+    This results in a quantitative
+    `Score <https://brain-score-core.readthedocs.io/en/latest/modules/metrics.html#brainscore_core.metrics.Score>`_
+    ranging from 0 (least brain-like) to 1 (most brain-like under this benchmark).
+    
+    :param model_identifier: the identifier for the model
+    :param benchmark_identifier: the identifier for the benchmark to test the model against
+    :return: a Score of how brain-like the candidate model is under this benchmark. The score is normalized by
+        this benchmark's ceiling such that 1 means the model matches the data to ceiling level.
+    """
+    return wrap_score(__file__,
+                      model_identifier=model_identifier, benchmark_identifier=benchmark_identifier,
+                      score_function=_run_score, conda_active=conda_active)

brainscore_language-2.1.0/brainscore_language/__main__.py

@@ -0,0 +1,14 @@
+from pathlib import Path
+
+import fire
+
+from brainscore_language import score as _score_function
+
+
+def score(model_identifier: str, benchmark_identifier: str, conda_active: bool=False):
+    result = _score_function(model_identifier, benchmark_identifier, conda_active)
+    print(result)  # print instead of return because fire has issues with xarray objects
+
+
+if __name__ == '__main__':
+    fire.Fire()

brainscore_language-2.1.0/brainscore_language/artificial_subject.py

@@ -0,0 +1,140 @@
+from typing import Dict, List, Union
+
+from brainio.assemblies import DataAssembly
+
+
+class ArtificialSubject:
+    def identifier(self) -> str:
+        """
+        The unique identifier for this model.
+
+        :return: e.g. `'glove'`, or `'distilgpt2'`
+        """
+        raise NotImplementedError()
+
+    class Task:
+        """ task to perform """
+        # ideally we would define these as enums (also for RecordingTarget and RecordingType)
+        # but accessing enums gives weird Enum objects whereas this setup just gives a string
+
+        next_word = 'next_word'
+        """ 
+        Predict the next word from the preceding context. Output a :class:`~brainio.assemblies.BehavioralAssembly` with 
+        next-word predictions as the values and preceding stimulus in the `stimulus` coordinate.
+
+        Example:
+
+        Calling `digest_text(text = 'the quick brown')` could output 
+
+        .. code-block:: python
+
+           {'behavior': <xarray.BehavioralAssembly (presentation: 1)>
+                        array(['fox']), # the actual next word
+                        Coordinates:
+                          * presentation  (presentation) MultiIndex
+                          - stimulus      (presentation) object 'the quick brown'
+                          - stimulus_id   (presentation) int64 0}
+
+        Example:
+
+        Calling `digest_text(text = ['the quick brown', 'fox jumps', 'over the lazy'])` could output 
+
+        .. code-block:: python
+
+           {'behavior': <xarray.BehavioralAssembly (presentation: 3)>
+                        array(['fox', 'over', 'dog']), # the actual next words
+                        Coordinates:
+                          * presentation  (presentation) MultiIndex
+                          - stimulus      (presentation) object 'the quick brown' 'fox jumps', 'over the lazy'
+                          - stimulus_id   (presentation) int64 0 1 2}
+        """
+
+        reading_times = 'reading_times'
+        """ 
+        Output how long it took to read a given text, in milliseconds. 
+        Output a :class:`~brainio.assemblies.BehavioralAssembly` with reading times in milliseconds as the values and 
+        text in the `stimulus` coordinate.
+
+        Example:
+
+        Calling `digest_text(text = 'the quick brown')` could output 
+
+        .. code-block:: python
+
+           {'behavior': <xarray.BehavioralAssembly (presentation: 1)>
+                        array([329.15]), # reading time in milliseconds
+                        Coordinates:
+                          * presentation  (presentation) MultiIndex
+                          - stimulus      (presentation) object 'the quick brown'
+                          - stimulus_id   (presentation) int64 0}
+
+        Example:
+
+        Calling `digest_text(text = ['the quick brown', 'fox jumps', 'over the lazy'])` could output 
+
+        .. code-block:: python
+
+           {'behavior': <xarray.BehavioralAssembly (presentation: 3)>
+                        array([329.15, 337.53, 341.13]), # reading times in milliseconds
+                        Coordinates:
+                          * presentation  (presentation) MultiIndex
+                          - stimulus      (presentation) object 'the quick brown' 'fox jumps', 'over the lazy'
+                          - stimulus_id   (presentation) int64 0 1 2}
+        """
+
+    def start_behavioral_task(self, task: Task):
+        """
+        Instruct the subject to output behavior when queried with
+        :meth:`~brainscore_language.artificial_subject.ArtificialSubject.digest_text`. Calling
+        :meth:`~brainscore_language.artificial_subject.ArtificialSubject.digest_text` will then output a
+        :class:`~brainio.assemblies.BehavioralAssembly` in the `'behavior'` part of the output dict.
+
+        :param task: Which :class:`~brainscore_language.artificial_subject.ArtificialSubject.Task` to use.
+        """
+        raise NotImplementedError()
+
+    class RecordingTarget:
+        """ location to record from """
+        language_system = "language_system"
+        language_system_left_hemisphere = "language_system_left_hemisphere"
+        language_system_right_hemisphere = "language_system_right_hemisphere"
+
+    class RecordingType:
+        """ method of recording """
+
+        fMRI = "fMRI"
+        """ functional magnetic resonance imaging """
+        ECoG = "ECoG"
+        """ Electrocorticography """
+
+    def start_neural_recording(self, recording_target: RecordingTarget, recording_type: RecordingType):
+        """
+        Begin neural recordings from this subject. Calling
+        :meth:`~brainscore_language.artificial_subject.ArtificialSubject.digest_text` will then output a
+        :class:`~brainio.assemblies.NeuroidAssembly` in the `'neural'` part of the output dict.
+
+        :param recording_target: where in the language-involved parts of the brain to record from.
+        :param  recording_type: What recording method should be used.
+        """
+        raise NotImplementedError()
+
+    def digest_text(self, text: Union[str, List[str]]) -> Dict[str, DataAssembly]:
+        """
+        :param text: text to pass to the subject, either a single string (e.g. `"the quick brown fox jumped"`),
+            or a list of strings (e.g. `["the quick", "brown fox jumped"]`)
+        :return: a dictionary mapping from `'behavior'` and `'neural'` to a :class:`~brainio.assemblies.DataAssembly`,
+            if the subject was instructed to
+            :meth:`~brainscore_language.artificial_subject.ArtificialSubject.start_behavioral_task` and/or
+            :meth:`~brainscore_language.artificial_subject.ArtificialSubject.start_neural_recording` respectively.
+            The :class:`~brainio.assemblies.DataAssembly` always contains a `presentation` dimension, corresponding to
+            the presentation of text stimuli (including a `stimulus` coordinate for the text,
+            and a `stimulus_id` coordinate that uniquely identifies each stimulus).
+            Values along the `presentation` dimension are ordered in the same way as the input.
+            The `behavior` output has no other dimension, while the `neural` output has a `neuroid` dimension
+            (including a `neuroid_id` coordinate to uniquely identify each recording unit), and sometimes a
+            `time_bin` dimension (including `time_bin_start` and `time_bin_end`).
+            Note that model state is not saved between every time that `digest_text` is called. For keeping states,
+            between different inputs, `text` should be a list.
+
+        """
+        raise NotImplementedError()

brainscore_language-2.1.0/brainscore_language/benchmark_helpers/__init__.py

@@ -0,0 +1,47 @@
+import numpy as np
+from typing import List
+
+from brainio.assemblies import walk_coords, array_is_element, DataAssembly
+
+
+def ci_error(samples, center, confidence=.95):
+    low, high = 100 * ((1 - confidence) / 2), 100 * (1 - ((1 - confidence) / 2))
+    confidence_below, confidence_above = np.nanpercentile(samples, low), np.nanpercentile(samples, high)
+    confidence_below, confidence_above = center - confidence_below, confidence_above - center
+    return confidence_below, confidence_above
+
+
+def manual_merge(*elements: List[DataAssembly], on='neuroid') -> DataAssembly:
+    """
+    Manually merge a set of assemblies where xarray's automated merge might fail.
+    This function likely covers only covers a small number of use-cases, and should thus be used with caution.
+    """
+    dims = elements[0].dims
+    assert all(element.dims == dims for element in elements[1:])
+    merge_index = dims.index(on)
+    # the coordinates in the merge index should have the same keys
+    assert _coords_match(elements, dim=on,
+                         match_values=False), f"coords in {[element[on] for element in elements]} do not match"
+    # all other dimensions, their coordinates and values should already align
+    for dim in set(dims) - {on}:
+        assert _coords_match(elements, dim=dim,
+                             match_values=True), f"coords in {[element[dim] for element in elements]} do not match"
+    # merge values without meta
+    merged_values = np.concatenate([element.values for element in elements], axis=merge_index)
+    # piece together with meta
+    result = type(elements[0])(merged_values, coords={
+        **{coord: (dims, values)
+           for coord, dims, values in walk_coords(elements[0])
+           if not array_is_element(dims, on)},
+        **{coord: (dims, np.concatenate([element[coord].values for element in elements]))
+           for coord, dims, _ in walk_coords(elements[0])
+           if array_is_element(dims, on)}}, dims=elements[0].dims)
+    return result
+
+
+def _coords_match(elements, dim, match_values=False):
+    """ Helper method for `manual_merge` """
+    first_coords = [(key, tuple(value)) if match_values else key for _, key, value in walk_coords(elements[0][dim])]
+    other_coords = [[(key, tuple(value)) if match_values else key for _, key, value in walk_coords(element[dim])]
+                    for element in elements[1:]]
+    return all(tuple(first_coords) == tuple(coords) for coords in other_coords)

brainscore_language-2.1.0/brainscore_language/benchmarks/blank2014/__init__.py

@@ -0,0 +1,4 @@
+from brainscore_language import benchmark_registry
+from .benchmark import Blank2014Linear
+
+benchmark_registry['Blank2014-linear'] = Blank2014Linear

brainscore_language-2.1.0/brainscore_language/benchmarks/blank2014/benchmark.py

@@ -0,0 +1,50 @@
+import xarray as xr
+
+from brainscore_core.benchmarks import BenchmarkBase
+from brainscore_core.metrics import Score
+from brainscore_language import load_dataset, load_metric
+from brainscore_language.artificial_subject import ArtificialSubject
+from brainscore_language.benchmarks.blank2014.ceiling import ExtrapolationCeiling
+from brainscore_language.data.blank2014 import BIBTEX
+from brainscore_language.utils.ceiling import ceiling_normalize
+
+
+class Blank2014Linear(BenchmarkBase):
+    """
+    Evaluate model ability to predict neural activity in human language system functional regions of interest (fROIs)
+    in response to natural stories, recorded by Blank et al. 2014.
+    Alignment of neural activity between model and human subjects is evaluated via cross-validated linear predictivity.
+
+    This benchmark builds off the Blank2014 benchmark introduced in Schrimpf et al. 2021
+    (https://www.pnas.org/doi/10.1073/pnas.2105646118), but requires the model to have committed to neural readouts
+    (e.g. "layer 41 corresponds to the language system"), rather than testing every layer separately.
+    """
+
+    def __init__(self):
+        self.data = load_dataset('Blank2014.fROI')
+        self.metric = load_metric('linear_pearsonr')
+        ceiler = ExtrapolationCeiling()
+        ceiling = ceiler(assembly=self.data, metric=self.metric)
+        super(Blank2014Linear, self).__init__(
+            identifier='Blank2014-linear',
+            version=1,
+            parent='neural_language',
+            ceiling=ceiling,
+            bibtex=BIBTEX)
+
+    def __call__(self, candidate: ArtificialSubject) -> Score:
+        candidate.start_neural_recording(recording_target=ArtificialSubject.RecordingTarget.language_system,
+                                         recording_type=ArtificialSubject.RecordingType.fMRI)
+        stimuli = self.data['stimulus']
+        stories = self.data['story'].values
+        predictions = []
+        for story in sorted(set(stories)):  # go over individual stories, sorting to keep consistency across runs
+            story_indexer = [stimulus_story == story for stimulus_story in stories]
+            story_stimuli = stimuli[story_indexer]
+            story_predictions = candidate.digest_text(story_stimuli.values)['neural']
+            story_predictions['stimulus_id'] = 'presentation', story_stimuli['stimulus_id'].values
+            predictions.append(story_predictions)
+        predictions = xr.concat(predictions, dim='presentation')
+        raw_score = self.metric(predictions, self.data)
+        score = ceiling_normalize(raw_score, self.ceiling)
+        return score