returnn 1.20240620.201955__tar.gz → 1.20240621.145554__tar.gz

{returnn-1.20240620.201955 → returnn-1.20240621.145554}/CHANGELOG.md

@@ -6,6 +6,139 @@ or any changes which could potentially break or change the behavior of existing
 This is intentionally kept short. For a full change log, just see the Git log.
 
 
+## 2024-06-07: `VariableDataset`
+
+Custom subdataset per subepoch based on user-provided function.
+Can be useful for advanced training pipeline
+where you create some HDFs on-the-fly (e.g. for alignments)
+and then want to load them in later epochs.
+
+## 2024-05-28: [`DistributeFilesDataset`](https://github.com/rwth-i6/returnn/blob/master/returnn/datasets/distrib_files.py) ([PR #1521](https://github.com/rwth-i6/returnn/pull/1521), [issue #1519](https://github.com/rwth-i6/returnn/issues/1519))
+
+`DistributeDataset` together with `FileCache`
+allows to train on very large datasets which do not fit on the local disk.
+`DistributeDataset` operates on a list of files,
+and for each sub-epoch will only select a subset of the files,
+and `FileCache` will cache the files locally.
+
+It also was specifically designed with distributed training in mind.
+The distributed random_seed_offset method can be used,
+but sharding is also supported ([PR #1538](https://github.com/rwth-i6/returnn/pull/1538)).
+
+## 2023-11-09: `LearningRateControl` saves more meta info in learning-rate-file
+
+Like effective learning rate (after `dynamic_learning_rate`),
+training step, GPU, RETURNN version, etc.
+(Was extended a bit over time.)
+
+## 2024-01-09: [Train proc manager](https://github.com/rwth-i6/returnn/blob/master/returnn/util/train_proc_manager.py)
+
+Auto restart RETURNN on crashes under certain conditions
+(e.g. it must have trained at least one epoch successfully since the most recent restart).
+
+## 2023-12-30: PyTorch handle OOM for forwarding, auto-split batch
+
+This went through several iterations of approaches,
+stumbling through a number of CPython and PyTorch bugs,
+e.g. [CPython #113939](https://github.com/python/cpython/issues/113939),
+[PyTorch #18853](https://github.com/pytorch/pytorch/issues/18853),
+[PyTorch #27600](https://github.com/pytorch/pytorch/issues/27600).
+
+## 2023-12-23: PyTorch distributed training with param averaging
+
+In the `torch_distributed` config dict: Set `"reduce_type": "param"` and `"param_sync_step": ...`.
+
+## 2023-10-24: [`watch_memory`](https://github.com/rwth-i6/returnn/blob/master/returnn/util/watch_memory.py): watches memory of all procs
+
+## 2023-10-03: RETURNN frontend (RF) native helpers ([PR #1403](https://github.com/rwth-i6/returnn/pull/1403))
+
+## 2023-06-09: PyTorch distributed training ([PR #1335](https://github.com/rwth-i6/returnn/pull/1335), [issue #1332](https://github.com/rwth-i6/returnn/issues/1332))
+
+`torch_distributed` config setting.
+Using the official PyTorch `DistributedDataParallel`, i.e. synchronized accumulated gradients.
+Each worker uses a different `random_seed_offset` for the dataset.
+
+## 2023-05-15: PyTorch automatic mixed precision (AMP) support ([PR #1322](https://github.com/rwth-i6/returnn/pull/1322))
+
+## 2023-04-03: PyTorch `preload_from_files` support ([PR #1292](https://github.com/rwth-i6/returnn/pull/1292))
+
+## 2023-03-26: [`MultiProcDataset`](https://github.com/rwth-i6/returnn/blob/master/returnn/datasets/multi_proc.py)
+
+## 2023-02-24: Make [`Tensor` and `Dim`](https://returnn.readthedocs.io/en/latest/getting_started/data.html) backend independent ([PR #1261](https://github.com/rwth-i6/returnn/pull/1261), [issue #1165](https://github.com/rwth-i6/returnn/issues/1165))
+
+* Rename `Data` to `Tensor`, `DimensionTag` to `Dim`.
+* Before, in our `Tensor`, the `placeholder` (now `raw_tensor`) was either None (as a template)
+  or a TensorFlow tensor (`tf.Tensor`).
+  Now it can support any raw tensor type.
+* Now `Tensor` and `Dim` are moved to `returnn.tensor`.
+
+## 2023-02-20: RETURNN frontend (RF) ([issue #1120](https://github.com/rwth-i6/returnn/issues/1120), [issue #1264](https://github.com/rwth-i6/returnn/issues/1264))
+
+Modern alternative to the network dictionary to define models.
+Using Python code to define the network,
+very similar to how it is done in PyTorch or Keras or Flax.
+
+This evolved from [`returnn_common.nn`](https://github.com/rwth-i6/returnn_common/tree/main/nn) ([example](https://github.com/rwth-i6/returnn_common/wiki/RETURNN-example-config)),
+which provided already a very similar API.
+But now, we build it such that we support multiple backends.
+Specifically, the current supported (or planned) backends:
+
+* PyTorch (fully supported)
+* RETURNN network dictionary (TensorFlow) (fully supported)
+  (copied the `returnn_common.nn` code)
+  (this might be deprecated in the future)
+* TensorFlow (directly) (mostly supported)
+* NumPy (partially supported)
+* JAX (planned)
+
+## 2023-02-03: Use `black`, drop Python 2 support ([PR #1255](https://github.com/rwth-i6/returnn/pull/1255), [issue #487](https://github.com/rwth-i6/returnn/issues/487), [issue #1158](https://github.com/rwth-i6/returnn/issues/1158))
+
+## 2022-10-24: Remove Theano backend ([PR #1164](https://github.com/rwth-i6/returnn/pull/1164))
+
+## 2022-09-12: PyTorch backend started ([issue #1120](https://github.com/rwth-i6/returnn/issues/1120))
+
+This evolved over time.
+It was planned from the beginning
+to support pure PyTorch models defined by the user
+but also RETURNN frontend (RF) models.
+
+## 2022-04-24: TF eager execution initial support
+
+## 2022-02-11: TF loss auto-flatten optimization ([PR #906](https://github.com/rwth-i6/returnn/pull/906))
+
+## 2021-09-12: TF generalized attention, `CumConcatLayer` ([PR #589](https://github.com/rwth-i6/returnn/pull/589), [issue #391](https://github.com/rwth-i6/returnn/issues/391))
+
+Generalizes `SelfAttentionLayer` to allow for more custom variants.
+The difficulties were to support this when being inside a `RecLayer`
+and then when the optimization would move it outside the loop.
+For this, we introduced `CumConcatLayer`.
+[Example config for decoder self-attention](https://github.com/rwth-i6/returnn/issues/391#issuecomment-917517032).
+This can also be used just in the encoder, i.e. outside a `RecLayer` anyway
+via `ReinterpretDataLayer` to create a new dim tag.
+[Example config for encoder self-attention](https://github.com/rwth-i6/returnn/issues/391#issuecomment-919873563).
+
+## 2021-08-25: Explicit `Data` dimension tags ([PR #579](https://github.com/rwth-i6/returnn/pull/579))
+
+`Data` (later called `Tensor`) has `dim_tags` (later called `dims`)
+to describe the full shape, i.e. the dims of each axis.
+These are `DimensionTag` (later `Dim`) objects.
+Before this change, we already had dim tags but only for dynamic dims.
+Now they are consistently used for all dims.
+This makes everything more consistent and more in line
+with other named tensors / named dimensions frameworks.
+
+## 2021-06-11: [Behavior versions](https://returnn.readthedocs.io/en/latest/configuration_reference/behavior_version.html) ([PR #534](https://github.com/rwth-i6/returnn/pull/534), [issue #508](https://github.com/rwth-i6/returnn/issues/508))
+
+Setting `behavior_version` in config control the behavior of RETURNN
+and allows to update bad/buggy/broken behavior without changing behavior for existing setups.
+
+## 2021-06-04: Start of [`returnn_common.nn`](https://github.com/rwth-i6/returnn_common/tree/main/nn)
+
+Allows to define the RETURNN network dictionary using a more modern Python API,
+very similar to PyTorch or Keras.
+[Example](https://github.com/rwth-i6/returnn_common/wiki/RETURNN-example-config).
+Note that this later got merged into RETURNN frontend (RF).
+
 ## 2021-03-18: Subnetwork sub layer can be independent ([#473](https://github.com/rwth-i6/returnn/pull/473))
 
 This has an effect on recurrent subnetworks.
@@ -103,25 +236,25 @@ This will show the same information as before, but much more compact,
 and also in addition the dimension tags (`DimensionTag`),
 which also got improved in many further cases.
 
-## 2019-08-07: overlay nets (`extra_nets`)
+## 2019-08-07: Overlay nets (`extra_nets`)
 
 You can have e.g. multiple additional networks which redefine
 existing layers (they would automatically share params),
 which can use different flags (e.g. enable the search flag).
 
-## 2019-07: multiple stochastic (latent) variables
+## 2019-07: Multiple stochastic (latent) variables
 
 It was designed to support this from the very beginning,
 but the implementation was never fully finished for this.
 Now examples like hard attention work.
 
-## 2019-05: better support for RETURNN as a framework
+## 2019-05: Better support for RETURNN as a framework
 
 `pip install returnn`, and then `import returnn`.
 
-## 2019-03-29: remove hard Theano dependency
+## 2019-03-29: Remove hard Theano dependency
 
-## 2019-03-24 and ongoing: automatic linter checks
+## 2019-03-24 and ongoing: Automatic linter checks
 
 Currently pylint and PyCharm inspection checks automatically run in Travis.
 Both have some false positives, but so far the PyCharm inspections seems much more sane.
@@ -180,16 +313,16 @@ and then later `SplitBatchTimeLayer` to get the time-axis back, it was likely in
 
 ## 2019-01-30: video: RETURNN overview
 
-## 2018-08: multi-GPU support via [Horovod](https://github.com/horovod/horovod)
+## 2018-08: Multi-GPU support via [Horovod](https://github.com/horovod/horovod)
 
-## 2017-05: flexible `RecLayer`, encoder-decoder attention, beam search (Albert Zeyer)
+## 2017-05: Flexible `RecLayer`, encoder-decoder attention, beam search (Albert Zeyer)
 
-## 2016-12: start on [TensorFlow](https://www.tensorflow.org/) support (Albert Zeyer)
+## 2016-12: Start on [TensorFlow](https://www.tensorflow.org/) support (Albert Zeyer)
 
 Initial working support already finished within that month.
 TF 0.12.0.
 
-## 2015-07: fast CUDA LSTM kernel (Paul Voigtlaender)
+## 2015-07: Fast CUDA LSTM kernel (Paul Voigtlaender)
 ## 2015-03: `SprintDataset`, interface to [RASR](https://www-i6.informatik.rwth-aachen.de/rwth-asr/) (Albert Zeyer)
 ## 2015-01: Albert Zeyer joined
 ## ~2013-2014 (?): Patrick Doetsch started the project (Theano)

{returnn-1.20240620.201955/returnn.egg-info → returnn-1.20240621.145554}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: returnn
-Version: 1.20240620.201955
+Version: 1.20240621.145554
 Summary: The RWTH extensible training framework for universal recurrent neural networks
 Home-page: https://github.com/rwth-i6/returnn/
 Author: Albert Zeyer

returnn-1.20240621.145554/_setup_info_generated.py

@@ -0,0 +1,2 @@
+version = '1.20240621.145554'
+long_version = '1.20240621.145554+git.5fd7cbc'

{returnn-1.20240620.201955 → returnn-1.20240621.145554}/returnn/util/file_cache.py

@@ -55,8 +55,9 @@ class FileCache:
         *,
         cache_directory: str = "$TMPDIR/$USER/returnn/file_cache",
         cleanup_files_always_older_than_days: float = 31.0,
-        cleanup_files_wanted_older_than_days: float = 7.0,
+        cleanup_files_wanted_older_than_days: float = 1.0,
         cleanup_disk_usage_wanted_free_ratio: float = 0.2,  # try to free at least 20% disk space
+        cleanup_disk_usage_wanted_multiplier: float = 2.0,  # try to free 2x the required space for a file
         num_tries: int = 3,  # retry twice by default
     ):
         """
@@ -66,12 +67,16 @@ class FileCache:
         :param cleanup_files_wanted_older_than_days: if cleanup_disk_usage_wanted_free_ratio not reached,
             cleanup files older than this.
         :param cleanup_disk_usage_wanted_free_ratio: try to free at least this ratio of disk space.
+        :param cleanup_disk_usage_wanted_multiplier: when making space for a new file, try to free at
+            least this times as much space.
         :param num_tries: how many times to try caching a file before giving up
         """
         self.cache_directory = expand_env_vars(cache_directory)
         self._cleanup_files_always_older_than_days = cleanup_files_always_older_than_days
         self._cleanup_files_wanted_older_than_days = cleanup_files_wanted_older_than_days
         self._cleanup_disk_usage_wanted_free_ratio = cleanup_disk_usage_wanted_free_ratio
+        assert cleanup_disk_usage_wanted_multiplier >= 1.0
+        self._cleanup_disk_usage_wanted_multiplier = cleanup_disk_usage_wanted_multiplier
         self._touch_files_thread = _TouchFilesThread(cache_base_dir=self.cache_directory)
         self._touch_files_thread.start()
         self._recent_full_cleanup_time = float("-inf")
@@ -143,7 +148,8 @@ class FileCache:
             return
         disk_usage = shutil.disk_usage(self.cache_directory)
         want_free_space_size = max(
-            need_at_least_free_space_size, int(self._cleanup_disk_usage_wanted_free_ratio * disk_usage.total)
+            int(self._cleanup_disk_usage_wanted_multiplier * need_at_least_free_space_size),
+            int(self._cleanup_disk_usage_wanted_free_ratio * disk_usage.total),
         )
         # If we have enough free space, and we did a full cleanup recently, we don't need to do anything.
         if want_free_space_size <= disk_usage.free and time.monotonic() - self._recent_full_cleanup_time < 60 * 10:
@@ -282,24 +288,24 @@ class FileCache:
         """
         Copy the file to the cache directory.
         """
-        if self._check_existing_copied_file_maybe_cleanup(src_filename, dst_filename):
-            os.utime(dst_filename, None)  # touch
-            return
-
         # Create dirs.
         dst_dir = os.path.dirname(dst_filename)
         os.makedirs(dst_dir, exist_ok=True)
 
         # Copy the file, while holding a lock. See comment on lock_timeout above.
-        with LockFile(directory=self.cache_directory, name="dir.lock", lock_timeout=self._lock_timeout) as lock:
+        with LockFile(
+            directory=dst_dir, name=os.path.basename(dst_filename) + ".lock", lock_timeout=self._lock_timeout
+        ) as lock:
             # Maybe it was copied in the meantime, while waiting for the lock.
             if self._check_existing_copied_file_maybe_cleanup(src_filename, dst_filename):
+                print(f"FileCache: using existing file {dst_filename}")
+                os.utime(dst_filename, None)  # touch
                 return
 
             print(f"FileCache: Copy file {src_filename} to cache")
 
-            # Make sure we have enough disk space.
-            self.cleanup(need_at_least_free_space_size=os.stat(src_filename).st_size)
+            # Make sure we have enough disk space, st_size +1 due to _copy_with_prealloc
+            self.cleanup(need_at_least_free_space_size=os.stat(src_filename).st_size + 1)
 
             dst_tmp_filename = dst_filename + ".copy"
             if os.path.exists(dst_tmp_filename):

{returnn-1.20240620.201955 → returnn-1.20240621.145554/returnn.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: returnn
-Version: 1.20240620.201955
+Version: 1.20240621.145554
 Summary: The RWTH extensible training framework for universal recurrent neural networks
 Home-page: https://github.com/rwth-i6/returnn/
 Author: Albert Zeyer

returnn-1.20240620.201955/_setup_info_generated.py

@@ -1,2 +0,0 @@
-version = '1.20240620.201955'
-long_version = '1.20240620.201955+git.7a0206a'