dimcat 3.2.0__tar.gz → 3.4.0__tar.gz

{dimcat-3.2.0 → dimcat-3.4.0}/.pre-commit-config.yaml

@@ -2,7 +2,7 @@ exclude: '^docs/conf.py'
 
 repos:
 - repo: https://github.com/pre-commit/pre-commit-hooks
-  rev: v4.4.0
+  rev: v5.0.0
   hooks:
   - id: trailing-whitespace
   - id: check-ast
@@ -32,12 +32,13 @@ repos:
   hooks:
   - id: seed-isort-config
 - repo: https://github.com/pycqa/isort
-  rev: 5.12.0
+  rev: 6.0.1
   hooks:
   - id: isort
+    args: ["--profile", "black", "--filter-files"]
 
 - repo: https://github.com/ambv/black
-  rev: 23.9.0
+  rev: 25.1.0
   hooks:
     - id: black
       language_version: python3.10
@@ -50,7 +51,7 @@ repos:
 #     additional_dependencies: [black]
 
 - repo: https://github.com/PyCQA/flake8
-  rev: 6.1.0
+  rev: 7.1.2
   hooks:
   - id: flake8
     args:

{dimcat-3.2.0 → dimcat-3.4.0}/CONTRIBUTING.rst

@@ -132,7 +132,6 @@ Clone the repository
 
 #. Install |pre-commit|_::
 
-    pip install pre-commit
     pre-commit install
 
    ``DiMCAT`` comes with a lot of hooks configured to automatically help the
@@ -202,57 +201,7 @@ Submit your contribution
 DiMCAT architecture
 -------------------
 
-1. The library is called DiMCAT and has three high-level objects:
-
-   a. :class:`~.DimcatObject` ("object"): the base class for all objects that manages object creation and serialization and subclass registration.
-      The DimcatObject class has a class attribute called _registry that is a dictionary of all subclasses of DimcatObject.
-      Each DimcatObject has a nested class called Schema that inherits from DimcatSchema.
-   #. :class:`~.DimcatSchema` ("schema"): the base class for all nested Schema classes, inheriting from marshmallow.Schema.
-      The Schema defines the valid values ranges for all attributes of the DimcatObject and how to serialize and deserialize them.
-   #. :class:`~.DimcatConfig` ("config"): a DimcatObject that can represent a subset of the attributes of another DimcatObject and instantiate it using the .create() method.
-      It derives from MutableMapping and used for communicating about and checking the compatibility of DimcatObjects.
-
-#. The nested Schema corresponding to each DimcatObject is instantiated as a singleton and can be retrieved via the class attribute :attr:`~.DimcatObject.schema`.
-   Using this Schema, a DimcatObject can be serialized to and deserialized from:
-
-   a. a dictionary using the :meth:`~.DimcatObject.to_dict` and :meth:`~.DimcatObject.from_dict` methods.
-   #. a DimcatConfig object using the :meth:`~.DimcatObject.to_config` and :meth:`~.DimcatObject.from_config` methods.
-   #. a JSON string using the :meth:`~.DimcatObject.to_json` and :meth:`~.DimcatObject.from_json` methods.
-   #. a JSON file using the :meth:`~.DimcatObject.to_json_file` and :meth:`~.DimcatObject.from_json_file` methods.
-
-   In the following, by "serialized object" we mean its representation as a DimcatConfig if not otherwise specified.
-
-#. All objects that are neither a schema nor a config are one of the two following subclasses of DimcatObject:
-
-   a. :class:`~.Data`: a DimcatObject that represents a dataset, a subset of a dataset, or a an individual resource such as a dataframe.
-   #. :class:`~.PipelineStep`: a DimcatObject that accepts a Data object as input and returns a Data object as output.
-
-#. The principal Data object is called :class:`~.Dataset` and is the one that users will interact with the most.
-   The Dataset provides convenience methods that are equivalent to applying the corresponding PipelineStep.
-   Every PipelineStep applied to it will return a new Dataset that can be serialized and deserialized to re-start the pipeline from that point.
-   To that aim, every Dataset stores a serialization of the applied PipelineSteps and of the original Dataset that served as initial input.
-   This initial input is specified as a :class:`~.DimcatCatalog` which is a collection of :class:`DimcatPackages <.data.dataset.base.DimcatPackage>`,
-   each of which is a collection of :class:`DimcatResources <.data.resources.base.DimcatResource>`,
-   as defined by the `Frictionless Data specifications <https://frictionlessdata.io>`__.
-   The preferred structure of a DimcatPackage is a .zip and a datapackage.json file, where the former contains one or several .tsv files (resources) described in the latter.
-   Since the data that DiMCAT transforms and analyzes comes from very heterogeneous sources, each original corpus is pre-processed and stored as a `frictionless.Package <https://framework.frictionlessdata.io/docs/framework/package.html>`__ together with the metadata relevant for reproducing the pre-processing.
-#. It follows that the Dataset is mainly a container for :class:`DimcatResources <.data.resources.base.DimcatResource>` namely:
-
-   a. Facets, i.e. the resources described in the original datapackage.json. They aim to stay as faithful as possible to the original data, applying only mild standardization and normalization.
-      All Facet resources come with several columns that represent timestamps both in absolute and in musical time, allowing for the alignment of different corpora.
-      The `Frictionless resource <https://framework.frictionlessdata.io/docs/framework/resource.html>`__ descriptors listed in the datapackage.json contain both the column schema and the piece IDs that are present in each of the facets.
-   #. :class:`Features <.data.resources.features.Feature>`, i.e. resources derived from Facets by applying PipelineSteps. They are standardized objects that are requested by the PipelineSteps to compute statistics and visualizations.
-      To allow for straightforward serialization of the Dataset, all Feature resources are represented as a DimcatCatalog called `outputs`, which can be stored as .tsv files in one or several .zip files.
-
-#. A :class:`~.DimcatResource` functions similarly to the `frictionless.Resource <https://framework.frictionlessdata.io/docs/framework/resource.html>`__ that it wraps, meaning that it grants access to the metadata without having to load the dataframes into memory.
-   It can be instantiated in two different ways, either from a resource descriptor or from a dataframe.
-   At any given moment, the :attr:`~.DimcatResource.status` attribute returns an Enum value reflecting the availability and state of the/a dataframe.
-   When a Dataset is serialized, all dataframes from the outputs catalog that haven't been stored to disk yet are written into one or several .zip files so that they can be referenced by resource descriptors.
-#. One of the most important methods, used by most PipelineSteps, is :meth:`.Dataset.get_feature`, which accepts a Feature config and returns a Feature resource.
-   The Feature config is a :class:`~.DimcatConfig` that specifies the type of Feature to be returned and the parameters to be used for its computation. Furthermore, it is also used
-
-   a. to determine for each piece in every loaded DimcatPackage an Availability value, ranging from not available over available with heavy computation to available instantly.
-   #. to determine whether the Feature resource had already been requested and stored in the outputs catalog.
+See `Developers guide <https://dimcat.readthedocs.io/en/latest/manual/data.html>`__.
 
 
 Coding Conventions
@@ -267,21 +216,31 @@ Commit messages
 ``DiMCAT`` uses `Conventional Commits <https://www.conventionalcommits.org/>`__ to determine the next SemVer version number. Please make sure to prefix each
 message with one of:
 
-+-------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
-| Commit Type | Title                    | Description                                                                                                 | SemVer |
-+=============+==========================+=============================================================================================================+========+
-| `feat`      | Features                 | A new feature                                                                                               | MINOR  |
-| `fix`       | Bug Fixes                | A bug Fix                                                                                                   | PATCH  |
-| `docs`      | Documentation            | Documentation only changes                                                                                  | PATCH  |
-| `style`     | Styles                   | Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)      | PATCH  |
-| `refactor`  | Code Refactoring         | A code change that neither fixes a bug nor adds a feature                                                   | PATCH  |
-| `perf`      | Performance Improvements | A code change that improves performance                                                                     | PATCH  |
-| `test`      | Tests                    | Adding missing tests or correcting existing tests                                                           | PATCH  |
-| `build`     | Builds                   | Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)         | PATCH  |
-| `ci`        | Continuous Integrations  | Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs) | PATCH  |
-| `chore`     | Chores                   | Other changes that don't modify src or test files                                                           | PATCH  |
-| `revert`    | Reverts                  | Reverts a previous commit                                                                                   | PATCH  |
-+-------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
++---------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
+| Commit Type   | Title                    | Description                                                                                                 | SemVer |
++===============+==========================+=============================================================================================================+========+
+| ``feat``      | Features                 | A new feature                                                                                               | MINOR  |
++---------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
+| ``fix``       | Bug Fixes                | A bug Fix                                                                                                   | PATCH  |
++---------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
+| ``docs``      | Documentation            | Documentation only changes                                                                                  | PATCH  |
++---------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
+| ``style``     | Styles                   | Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)      | PATCH  |
++---------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
+| ``refactor``  | Code Refactoring         | A code change that neither fixes a bug nor adds a feature                                                   | PATCH  |
++---------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
+| ``perf``      | Performance Improvements | A code change that improves performance                                                                     | PATCH  |
++---------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
+| ``test``      | Tests                    | Adding missing tests or correcting existing tests                                                           | PATCH  |
++---------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
+| ``build``     | Builds                   | Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)         | PATCH  |
++---------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
+| ``ci``        | Continuous Integrations  | Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs) | PATCH  |
++---------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
+| ``chore``     | Chores                   | Other changes that don't modify src or test files                                                           | PATCH  |
++---------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
+| ``revert``    | Reverts                  | Reverts a previous commit                                                                                   | PATCH  |
++---------------+--------------------------+-------------------------------------------------------------------------------------------------------------+--------+
 
 In the case of breaking changes, which result in a new major version, please add a ``!`` after the type, e.g., ``refactor!:``.
 This type of commit message needs to come with a body, starting with ``BREAKING CHANGE:``, which explains in great detail everything

{dimcat-3.2.0 → dimcat-3.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: dimcat
-Version: 3.2.0
+Version: 3.4.0
 Summary: DIgital Musicology Corpus Analysis Toolkit
 Home-page: https://github.com/DCMLab/dimcat
 Author: Digital and Cognitive Musicology Lab @ École Polytechnique Fédérale de Lausanne

{dimcat-3.2.0 → dimcat-3.4.0}/docs/CHANGELOG.md

@@ -1,5 +1,51 @@
 # Changelog
 
+## [3.4.0](https://github.com/DCMLab/dimcat/compare/v3.3.0...v3.4.0) (2025-04-28)
+
+
+### Features
+
+* allows for retrieving pipeline steps via subscript ([f774987](https://github.com/DCMLab/dimcat/commit/f7749874f9c1fb930d7542637c11efae05ce7056))
+
+
+### Bug Fixes
+
+* adds default value column to Measures feature ([85f12c6](https://github.com/DCMLab/dimcat/commit/85f12c67435a2d5aa8c8d28de9ec90b859dea79b))
+* avoids an error due to missing columns ([efe6947](https://github.com/DCMLab/dimcat/commit/efe6947af215dd88c7c8321ca6243452e71772ab))
+
+
+### Documentation
+
+* extends docstring for FeatureDimensionSlicer ([cb897da](https://github.com/DCMLab/dimcat/commit/cb897da170c3b6d0b29cae12e6800b674837c11a))
+
+## [3.3.0](https://github.com/DCMLab/dimcat/compare/v3.2.0...v3.3.0) (2024-11-18)
+
+
+### Features
+
+* make_histogram() and make_violin_plot() ([ec89ce2](https://github.com/DCMLab/dimcat/commit/ec89ce2799d822f244aae11cbae5125202eee69f))
+
+
+### Bug Fixes
+
+* avoids duplicate column when applying stage criterion ([66aa952](https://github.com/DCMLab/dimcat/commit/66aa952980ec6161efbcae5ba68f46ea8b450fb0))
+* check column directly for the time being (fields are not yet updated automatically) ([aa3376c](https://github.com/DCMLab/dimcat/commit/aa3376c48c43180a63aae0e148a27561fffdcf22))
+* correct inner range index even if its start positions overlap those of the outer start mask ([eb45b17](https://github.com/DCMLab/dimcat/commit/eb45b1787fa3745fe63034126826d40bf0a0ce6b))
+* stop Dataset.iter_features() after iterating through all ([6035174](https://github.com/DCMLab/dimcat/commit/6035174c7289ca5bbb71f4454acb0ea039931776))
+* workaround for avoiding PermissionError on Windows when using tempfile ([e5f3823](https://github.com/DCMLab/dimcat/commit/e5f382392b5476e46f5b5816723812b440f6e950))
+
+
+### Documentation
+
+* adds developer's guide notebook ([4fcd554](https://github.com/DCMLab/dimcat/commit/4fcd554d1e60a514c1803fdd1afefc73c1c7516a))
+* Dataset ([7717f16](https://github.com/DCMLab/dimcat/commit/7717f1668a0fc091a485004463e72445c9dbc0cd))
+* Dataset for developers ([b52692d](https://github.com/DCMLab/dimcat/commit/b52692d09355b67cff495cf878a171a4a3cf76c0))
+* fixes table in CONTRIBUTING.rst ([813da3f](https://github.com/DCMLab/dimcat/commit/813da3fe7fcaa6c49a42baaa3939aa811c591d3e))
+* more work on the developers docs ([fed676b](https://github.com/DCMLab/dimcat/commit/fed676b64fbf089b9b9ee647cd54d2cf577ef0ee))
+* moves architecture primer to developers guide ([19c7373](https://github.com/DCMLab/dimcat/commit/19c737388e53e0da2722cf952feb7d3211231088))
+* removes pip install (pre-commit is part of the [docs] optional requirements) ([ba2258f](https://github.com/DCMLab/dimcat/commit/ba2258fe4d23d5838ae6374c9c9c1d2f2550d2ea))
+* several docstrings ([a735464](https://github.com/DCMLab/dimcat/commit/a7354648abfe1290b06d582d836d61e410657bc0))
+
 ## [3.2.0](https://github.com/DCMLab/dimcat/compare/v3.1.0...v3.2.0) (2024-01-30)
 
 

dimcat-3.4.0/docs/manual/developers.md

@@ -0,0 +1,250 @@
+---
+jupytext:
+  formats: ipynb,md:myst
+  text_representation:
+    extension: .md
+    format_name: myst
+    format_version: 0.13
+    jupytext_version: 1.16.4
+kernelspec:
+  display_name: dimcat
+  language: python
+  name: dimcat
+---
+
+# Developers guide
+
+This guide is an introduction into `DiMCAT's` code architecture. Users who want to contribute to `DiMCAT` are invited to refer to the [contribution guidelines](https://dimcat.readthedocs.io/en/latest/contributing.html) which contain coding conventions and instructions how to set up the development environment.
+
++++
+
+## Introduction
+
+The library is called DiMCAT and has three high-level objects:
+
+1. {class}`~.DimcatObject` ("object"): the base class for all objects that manages object creation and serialization and subclass registration.
+   The DimcatObject class has a class attribute called _registry that is a dictionary of all subclasses of DimcatObject.
+   Each DimcatObject has a nested class called {class}`~.DimcatObject.Schema` that inherits from {class}`~.DimcatSchema`.
+2. {class}`~.DimcatSchema` ("schema"): the base class for all nested Schema classes, inheriting from [marshmallow.Schema](https://marshmallow.readthedocs.io/en/stable/marshmallow.schema.html).
+   The Schema defines the valid values ranges for all attributes of the DimcatObject and how to serialize and deserialize them.
+3. {class}`~.DimcatConfig` ("config"): a DimcatObject that can represent a subset of the attributes of another DimcatObject and instantiate it using the {meth}`~.DimcatConfig.create` method.
+   It derives from MutableMapping and is used for communicating about and checking the compatibility of DimcatObjects.
+
+The three classes are defined in the `src\dimcat\base.py` module.
+
++++
+
+### Serializing `DiMCAT` objects
+
+The nested Schema corresponding to each DimcatObject is instantiated as a singleton and can be retrieved via the class attribute {attr}`~.DimcatObject.schema`.
+Using this Schema, a DimcatObject can be serialized to and deserialized from:
+
+1. a dictionary using the {meth}`~.DimcatObject.to_dict` and {meth}`~.DimcatObject.from_dict` methods.
+2. a DimcatConfig object using the {meth}`~.DimcatObject.to_config` and {meth}`~.DimcatObject.from_config` methods.
+3. a JSON string using the {meth}`~.DimcatObject.to_json` and {meth}`~.DimcatObject.from_json` methods.
+4. a JSON file using the {meth}`~.DimcatObject.to_json_file` and {meth}`~.DimcatObject.from_json_file` methods.
+
+Under the hood, methods 2-4 use method 1. In addition, `DiMCAT` has the following standalone functions to deserialize serialized DimcatObjects:
+
+1. {func}`~.deserialize_dict`
+2. {func}`~.deserialize_config`
+3. {func}`~.deserialize_json_str`
+4. {func}`~.deserialize_json_file`
+
+This is possible because each deserialized object includes a value for the field `dtype` specifying the object's class name from which the schema can be retrieved thanks to the class attribute {attr}`~.DimcatObject.schema`. Other functions that are relevant in this context are {func}`~.get_class` and {func}`~.get_schema` (see [](#class-registry))
+
+#### Example
+
+```{code-cell}
+import dimcat as dc
+cfg = dc.DimcatConfig("DimcatObject")
+obj = cfg.create()
+print("This object is a", type(obj))
+json_str = obj.to_json()
+obj_copy = dc.base.DimcatObject.from_json(json_str)
+another_copy = dc.deserialize_json_str(json_str)
+print(f"The two deserialized objects are equivalent: ", obj_copy == another_copy)
+print(f"The two deserialized objects are identical: ", obj_copy is another_copy)
+obj_copy # DimcatObject.__repr__() uses .to_dict() under the hood
+```
+
+#### Implementation
+
+The implementation is centered on two methods of the respective object's nested {class}`~.DimcatSchema` which derives from [marshmallow.Schema](https://marshmallow.readthedocs.io/en/stable/marshmallow.schema.html): [schema.dump()](https://marshmallow.readthedocs.io/en/stable/marshmallow.schema.html#marshmallow.schema.Schema.dump) and [schema.load()](https://marshmallow.readthedocs.io/en/stable/marshmallow.schema.html#marshmallow.schema.Schema.load). The former takes an object and returns a dictionary, whereas the latter takes a dictionary and returns an object. Correspondingly, {meth}`.DimcatObject.to_dict`  and {meth}`.DimcatObject.from_dict` retrieve the relevant schema singleton from {attr}`.DimcatObject.schema` to call these two methods respectively.
+
++++
+
+### Creating a new type of DimcatObject
+
+All objects in `DiMCAT` (except {class}`~.DimcatSchema`) inherit from {class}`~.DimcatObject`. Inheritance also concerns the nested schema class. Effectively, this means that if you subclass an existing object type without adding new initialization arguments, your new class can simply inherit its parent's `Schema` class and serialization will just work as described above. However, if you add a property, meaning that you will also need to add the corresponding initialization argument, you also need to include a nested `Schema` class which inherits from the parent's schema. Each property that is to be serialized needs to be declared as [marshmallow field](https://marshmallow.readthedocs.io/en/stable/marshmallow.fields.html) corresponding to the datatype.
+
+```{code-cell}
+from marshmallow import fields
+
+class NewType(dc.base.DimcatObject):
+
+    class Schema(dc.base.DimcatObject.Schema):
+        new_property = fields.Str()
+
+    def __init__(self, new_property: str, **kwargs):
+        super().__init__(**kwargs)
+        self.new_property = new_property
+
+
+new_obj = NewType("some string value")
+as_dict = new_obj.to_dict()
+new_obj_copy = dc.deserialize_dict(as_dict)
+new_obj_copy
+```
+
+In cases where an attribute should point to a DimcatObject (e.g. all {class}`~.Result` objects referencing the analyzed {class}`~.DimcatResource` via the {attr}`~.analyzed_resource` property), we can use the type {class}`~.DimcatObjectField` in the schema.
+
++++
+
+(class-registry)=
+### The class registry
+
+Every {class}`~.DimcatObject` comes with the attribute {attr}`~._registry` which is a dictionary mapping the names of all DimcatObjects to their classes.
+It is implemented using [__init_subclass__](https://docs.python.org/3/reference/datamodel.html#object.__init_subclass__).
+We don't need to interact directy with the registry thanks to the convenience function {func}`~.get_class` which takes the name of an object as a string and returns the respective class.
+In the code, this would typically look like this:
+
+```{code-cell}
+Constructor = dc.get_class("FeatureExtractor")
+feature_extractor = Constructor()
+```
+
+Schemas are not part of the registry. For retrieving a class's schema we can use `Constructor.schema` (building on the example) or the convenience function {func}`~.get_schema`.
+
+
+### Public and private methods
+
+The DiMCAT project differentiates between private methods whose names begin with `_` and public methods whose names don't.
+Semantically, public methods are those that users interact with and which therefore often perform additional checks, e.g. of user input;
+then, the public method calls the private method of the same name which performs the actual job.
+In most cases, subclasses override only private methods.
+
+#### Example
+
+For example, compare the public {meth}`.PipelineStep.process_dataset` with its private counterpart:
+
+```python
+    def _process_dataset(self, dataset: Dataset) -> Dataset:
+        """Apply this PipelineStep to a :class:`Dataset` and return a copy containing the output(s)."""
+        new_dataset = self._make_new_dataset(dataset)
+        self.fit_to_dataset(new_dataset)
+        # this is where subclasses create a new package and add it to the dataset
+        return new_dataset
+    
+    def process_dataset(self, dataset: Dataset) -> Dataset:
+        """Apply this PipelineStep to a :class:`Dataset` and return a copy containing the output(s)."""
+        self.check_dataset(dataset)
+        return self._process_dataset(dataset)
+```
+
++++
+
+## Two types of DimcatObjects
+
+All classes that are neither a schema nor a config inherit from one of the two following subclasses of DimcatObject:
+
+1. {class}`~.Data`: a DimcatObject that represents a dataset, a subset of a dataset, or an individual resource such as a dataframe.
+2. {class}`~.PipelineStep`: a DimcatObject that accepts a Data object as input and returns a Data object as output.
+
+They are organized in two packages, {mod}`dimcat.data` and {mod}`dimcat.steps`. Objects defined in {mod}`dimcat.steps` operate on objects defined in {mod}`dimcat.data` and can import from it, but not the other way around.
+In a few exceptional cases where Data objects need to actively use PipelineStep (which is the case, for example, for {meth}`.Dataset.extract_feature`), we circumvent circular imports by summoning them via {func}`~.get_class` (and not using type hints for the summoned object).
+
+### Data objects
+
+Data is organized into a hierarchical hierarchy of four objects (from top to bottom):
+
+* {class}`~.Dataset`, consisting of two catalogs, called `inputs` and `outputs`;
+* {class}`~.DimcatCatalog` ("catalog"), a collection of packages;
+* {class}`~.DimcatPackage` ("package"), a collection of resources;
+* {class}`~.DimcatResource` ("resource"), a wrapper around a dataframe.
+
+There are three main types of resources, namely
+
+* {class}`~.Facet`, a representation of score-related elements that represents the loaded data with minimal standardization (e.g. {class}`~.MuseScoreNotes`);
+* {class}`~.Feature`, some aspect extracted from a facet, standardized by ``Dimcat`` (e.g., {class}`~.Notes`);
+* {class}`~.Result`, a result of applying a {class}`~.PipelineStep` to a feature.
+
+``Dimcat`` cannot process facets directly; the relevant features need to be extracted first. Many PipelineSteps extract the required feature automatically.
+
+#### Dataset
+
+The principal Data object is the {class}`~.Dataset` and is the one that users usually interact with the most. Its three principal properties are:
+
+* {attr}`~.inputs`, an {class}`~.InputsCatalog`
+* {attr}`~.outputs`, an {class}`~.OutputsCatalog`
+* {attr}`~.pipeline`, a {class}`~.Pipeline` consisting of all previously applied {class}`PipelineSteps <~.PipelineStep>`.
+
+After applying a {class}`~.PipelineStep` to a {class}`~.Dataset`,
+its {attr}`~.outputs` MUST correspond to the result of applying the {attr}`~.pipeline` to {attr}`~.inputs`.
+A serialized Dataset is therefore suited for communicating results in a reproducible manner.
+Any {class}`~.PipelineStep` applied on a dataset will be performed on all eligible resources that the packages in {attr}`~.inputs` contain and result in a new dataset containing the relevant output packages/resources under {attr}`~.outputs`.
+
+
+Datasets are passive 'by nature', meaning that, in general, they are being manipulated by PipelineSteps or by the user.
+PipelineSteps process a Dataset by requesting one or several features using {meth}`.Dataset.get_feature`,
+ processing each {class}`~.Feature`, and adding the processed Feature(s) to the Dataset's OutputsCatalog. 
+However, in one case, the Dataset *does* play an active role, namely in the extraction of features from the InputsCatalog.
+When prompted with `.get_feature(F)` where `F` is some specification of a {class}`~.Feature`, the Dataset will
+
+* look up the feature in its OutputCatalog and return it if present,
+* call {meth}`.Dataset.extract_feature` otherwise and return its output.
+
+Since the actual extraction happens on the level of a single resource (a {class}`~.Facet` which names the feature among its {attr}`~.DimcatResource.extractable_features`), the latter case invokes the following call chain:
+
+* {meth}`.Dataset.extract_feature` calls
+* {meth}`.InputsCatalog.extract_feature` calls
+* {meth}`.Package.extract_feature` calls 
+* {meth}`.DimcatResource.extract_feature`.
+
+The Dataset applies all previously applied PipelineSteps to the thus extracted {class}`~.Feature`,
+ adds it to its {class}`~.OutputsCatalog` and appends the {class}`~.FeatureExtractor` to its {class}`~.Pipeline`.
+
+
++++
+
+#### DimcatCatalog
+
+As per the [Frictionless Data specifications](https://frictionlessdata.io), a catalog is a collection of packages. In ``Dimcat``, catalogs appear only in a single place: Every {class}`~.Dataset` consists of two {class}`DimcatCatalogs <.DimcatCatalog>`, namely an {class}`~.InputsCatalog` and an {class}`~.OutputsCatalog`.
+{attr}`~.inputs` includes all loaded datapackages, and {attr}`~.outputs` all (processed or unprocessed) features extracted from them, as well as all other 
+processing results. 
+
+
+#### DimcatPackage
+
+The preferred structure of a DimcatPackage is a ``.zip`` and a ``.datapackage.json`` file, where the former contains one or several ``.tsv`` files (resources) described in the latter.
+The ``.datapackage.json`` file follows the [Frictionless](https://frictionlessdata.io) Data specifications for packages and allows ``DiMCAT`` to know what's in a package without having to actually load the data from the ``.tsv`` files.
+
+Since the data that DiMCAT transforms and analyzes comes from very heterogeneous sources, each original corpus is pre-processed and stored as a [frictionless.Package](https://framework.frictionlessdata.io/docs/framework/package.html). This task is achieved by loaders. When your aim is to enable ``DiMCAT`` to load a new type of dataset, you will have to implement a new {class}`~.Loader`. Please approach the developers by [creating an issue](https://github.com/DCMLab/dimcat/issues).
+
+#### DimcatResource
+
+From all this follows that the Dataset is mainly a structured container for {class}`DimcatResources <.DimcatResource>`, of which there are three main types:
+
+1. {class}`Facets <.Facet>`, i.e. the resources described in the original datapackage.json. They aim to stay as faithful as possible to the original data, applying only mild standardization and normalization.
+   All Facet resources come with several columns that represent timestamps both in absolute and in musical time, allowing for the alignment of different corpora.
+2. {class}`Features <.Feature>`, i.e. resources derived from Facets explicitly (via {meth}`.Dataset.extract_feature`) or implicitly (by applying PipelineSteps which extract features automatically). They are standardized objects that are required by  PipelineSteps to compute statistics and visualizations.
+   To allow for straightforward serialization of the Dataset, all Feature resources are represented as a DimcatCatalog called {attr}`~.Dataset.outputs`.
+3. {class}`Results <.Result>`, i.e., resources that represent the results of applying an {class}`~.Analyzer` to a {class}`~.Feature`. They are equally stored in the {attr}`~.Dataset.outputs` catalog and come with the appropriate methods for plotting the results.
+
+Although [Frictionless resources](https://framework.frictionlessdata.io/docs/framework/resource.html) can be stored as individual ``.tsv`` files with their own ``.resource.json`` descriptors,
+ ``DiMCAT`` typically stores multiple resources as a datapackage, i.e., a ``.zip`` file containing one ``.tsv`` file per resource, accompanied with a single ``.datapackage.json`` file detailing the metadata for all of them.
+ In both cases, the descriptor specifies the column schema for each resource, allowing ``DiMCAT`` to "lazy-load" the data, meaning that the Dataframes are loaded into memory only the moment when they are actually needed (using {meth}`.DimcatResource.load`).
+
+A DimcatResource can be instantiated in two different ways, either from a descriptor or from a dataframe.
+At any given moment, the {attr}`~.DimcatResource.status` attribute returns an Enum value reflecting the availability and state of the/a dataframe. This is relevant for keeping datapackages stored on disk up-to-date. 
+When a Dataset is serialized, all dataframes from the outputs catalog that haven't been stored to disk yet are written into one or several .zip files so that they can be referenced by the updated descriptor(s).
+
+If you want to create a new type of DimcatResource, please inherit from the relevant subclass and refer to the docstrings of {class}`~.DimcatResource` in the module {mod}`.dc` for understanding how to use the class variables.
+
++++
+
+
+
+```{code-cell}
+
+```

{dimcat-3.2.0 → dimcat-3.4.0}/docs/manual/index.rst

@@ -8,3 +8,4 @@ Manual
    quick
    data
    errors
+   developers

{dimcat-3.2.0 → dimcat-3.4.0}/docs/manual/quick.md

@@ -5,7 +5,7 @@ jupytext:
     extension: .md
     format_name: myst
     format_version: 0.13
-    jupytext_version: 1.16.0
+    jupytext_version: 1.16.1
 kernelspec:
   display_name: dimcat
   language: python
@@ -16,7 +16,7 @@ kernelspec:
 
 ## Import dimcat and load data
 
-```{code-cell}
+```{code-cell} ipython3
 import dimcat as dc
 from dimcat.data import resources
 from dimcat.steps import analyzers, extractors, groupers
@@ -28,7 +28,7 @@ dataset
 
 ## Show metadata
 
-```{code-cell}
+```{code-cell} ipython3
 dataset.get_metadata()
 ```
 
@@ -38,7 +38,7 @@ dataset.get_metadata()
 
 Here we pass the extracted notes to the counter.
 
-```{code-cell}
+```{code-cell} ipython3
 notes = dataset.get_feature("notes")
 result = analyzers.Counter().process(notes)
 result.plot()
@@ -46,14 +46,14 @@ result.plot()
 
 The `FeatureExtractor` is added to the dataset's pipeline implicitly, but the `Counter` is not because it's applied only to the extracted feature:
 
-```{code-cell}
+```{code-cell} ipython3
 dataset
 ```
 
 The pitch-class distributions shown by `.plot()` correspond to the current **unit of analysis**, which defaults to the piece-level.
 Results also come with a second plotting method, `.plot_grouped()`. Since no groupers have been applied, the entire dataset is treated as a single group:
 
-```{code-cell}
+```{code-cell} ipython3
 result.plot_grouped()
 ```
 
@@ -61,7 +61,7 @@ result.plot_grouped()
 
 Here we pass the dataset to the counter.
 
-```{code-cell}
+```{code-cell} ipython3
 counter = analyzers.Counter(features="notes")
 analyzed_dataset = counter.process(dataset)
 analyzed_dataset.get_result().plot()
@@ -70,13 +70,13 @@ analyzed_dataset.get_result().plot()
 Applying an `Analyzer` to a `Dataset` yields an `AnalyzedDataset` that includes one `Result` resource per analyzed `Feature`.
 Both are to be found in the respective packages in the outputs catalog:
 
-```{code-cell}
+```{code-cell} ipython3
 analyzed_dataset
 ```
 
 ### Variant 3: Define a Pipeline with FeatureExtractor and Counter
 
-```{code-cell}
+```{code-cell} ipython3
 pipeline = dc.Pipeline([
     extractors.FeatureExtractor("notes"),
     analyzers.Counter()
@@ -96,7 +96,7 @@ Let's define a CustomPieceGrouper from random piece groups:
   this level to any processed Resource (provided it contains the grouped pieces). This changes the behaviour of the grouped resource,
   e.g. when plotting it.
 
-```{code-cell}
+```{code-cell} ipython3
 n_groups = 3
 n_members = 30
 
@@ -107,12 +107,12 @@ grouper
 ### Applying the grouper to the analysis result
 ```
 
-```{code-cell}
+```{code-cell} ipython3
 grouped_result = grouper.process(result)
 grouped_result
 ```
 
-```{code-cell}
+```{code-cell} ipython3
 grouped_result.plot_grouped()
 ```
 
@@ -120,7 +120,7 @@ As promised, the grouped result plots differently: Instead of showing pitch-clas
 (which we can still obtain by calling `.plot()`), it shows the pitch-class distributions for each of the groups.
 However, for closer inspection, the area of a circle is not ideal, so let's view it as a bar plot:
 
-```{code-cell}
+```{code-cell} ipython3
 grouped_result.make_bar_plot()
 ```
 
@@ -131,22 +131,22 @@ resources (to the `Notes` feature and to the `Counts` result) and to a dataset c
 Another way to achieve the same goal is by applying steps to data. Let's start with a fresh dataset and
 apply the grouper and the analyzer once more:
 
-```{code-cell}
+```{code-cell} ipython3
 D = dc.Dataset.from_package(package_path)
 analyzed_dataset = D.apply_step(grouper, counter)
 analyzed_dataset
 ```
 
-```{code-cell}
+```{code-cell} ipython3
 result = analyzed_dataset.get_result()
 result
 ```
 
-```{code-cell}
+```{code-cell} ipython3
 result.default_groupby
 ```
 
-```{code-cell}
+```{code-cell} ipython3
 analyzed_dataset.get_result().make_bar_plot()
 ```
 
@@ -158,14 +158,14 @@ or invalid values [are rejected](./errors.md#invalid-option).
 
 Any DimcatObject can be expressed as a config by calling its `.to_config()` method:
 
-```{code-cell}
+```{code-cell} ipython3
 config = counter.to_config()
 config
 ```
 
 Any config can be used to instantiate a DimcatObject:
 
-```{code-cell}
+```{code-cell} ipython3
 counter_copy = config.create()
 print(f"""The new object and the old object are
 equal: {counter == counter_copy}
@@ -174,7 +174,7 @@ identical: {counter is counter_copy}""")
 
 Wherever DiMCAT operates with configs, it also accepts dictionaries:
 
-```{code-cell}
+```{code-cell} ipython3
 step_configs = [
     dict(dtype="FeatureExtractor", features=[dict(dtype="Notes", format="FIFTHS")]),
     dict(dtype='CustomPieceGrouper', grouped_units=grouping),
@@ -184,7 +184,7 @@ pl = dc.Pipeline.from_step_configs(step_configs)
 pl
 ```
 
-```{code-cell}
+```{code-cell} ipython3
 resulting_dataset = pl.process(dataset)
 resulting_dataset.get_result().make_bar_plot()
 ```

{dimcat-3.2.0 → dimcat-3.4.0}/docs/tutorial.ipynb

@@ -499,7 +499,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.4"
+   "version": "3.10.12"
   },
   "toc": {
    "base_numbering": 1,