pytesdaqx 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (211) hide show
  1. pytesdaqx-0.1.0/MANIFEST.in +4 -0
  2. pytesdaqx-0.1.0/PKG-INFO +93 -0
  3. pytesdaqx-0.1.0/README.md +54 -0
  4. pytesdaqx-0.1.0/docs/developer/acquisition_pipeline.md +118 -0
  5. pytesdaqx-0.1.0/docs/developer/adding_hardware.md +113 -0
  6. pytesdaqx-0.1.0/docs/developer/architecture.md +142 -0
  7. pytesdaqx-0.1.0/docs/developer/configuration_pipeline.md +136 -0
  8. pytesdaqx-0.1.0/docs/developer/data_io.md +220 -0
  9. pytesdaqx-0.1.0/docs/developer/hardware_support.md +133 -0
  10. pytesdaqx-0.1.0/docs/developer/index.md +41 -0
  11. pytesdaqx-0.1.0/docs/developer/package_structure.md +101 -0
  12. pytesdaqx-0.1.0/docs/developer/release_process.md +90 -0
  13. pytesdaqx-0.1.0/docs/developer/testing.md +89 -0
  14. pytesdaqx-0.1.0/docs/index.md +31 -0
  15. pytesdaqx-0.1.0/docs/installation/index.md +96 -0
  16. pytesdaqx-0.1.0/docs/installation/rocky_linux_9.md +120 -0
  17. pytesdaqx-0.1.0/docs/installation/troubleshooting.md +77 -0
  18. pytesdaqx-0.1.0/docs/installation/ubuntu.md +110 -0
  19. pytesdaqx-0.1.0/docs/installation/validation.md +103 -0
  20. pytesdaqx-0.1.0/docs/installation/visa.md +87 -0
  21. pytesdaqx-0.1.0/docs/user/cli/acquisition.md +114 -0
  22. pytesdaqx-0.1.0/docs/user/cli/data_inspection.md +142 -0
  23. pytesdaqx-0.1.0/docs/user/cli/index.md +59 -0
  24. pytesdaqx-0.1.0/docs/user/cli/instrument_control.md +144 -0
  25. pytesdaqx-0.1.0/docs/user/cli/lab_diagnostics.md +65 -0
  26. pytesdaqx-0.1.0/docs/user/configuration/acquisition_config.md +227 -0
  27. pytesdaqx-0.1.0/docs/user/configuration/hardware_yaml.md +164 -0
  28. pytesdaqx-0.1.0/docs/user/configuration/index.md +65 -0
  29. pytesdaqx-0.1.0/docs/user/configuration/resolution.md +65 -0
  30. pytesdaqx-0.1.0/docs/user/configuration/run_setup_yaml.md +183 -0
  31. pytesdaqx-0.1.0/docs/user/configuration/scans.md +529 -0
  32. pytesdaqx-0.1.0/docs/user/configuration/temperature_controllers.md +266 -0
  33. pytesdaqx-0.1.0/docs/user/data_format.md +205 -0
  34. pytesdaqx-0.1.0/docs/user/quickstart.md +94 -0
  35. pytesdaqx-0.1.0/docs/user/terminology.md +85 -0
  36. pytesdaqx-0.1.0/docs/user/troubleshooting.md +66 -0
  37. pytesdaqx-0.1.0/docs/user/workflows.md +62 -0
  38. pytesdaqx-0.1.0/examples/README.md +103 -0
  39. pytesdaqx-0.1.0/examples/configuration/basic/acquisition_background.yaml +22 -0
  40. pytesdaqx-0.1.0/examples/configuration/basic/acquisition_didv_background.yaml +45 -0
  41. pytesdaqx-0.1.0/examples/configuration/basic/acquisition_notebook_demo.yaml +25 -0
  42. pytesdaqx-0.1.0/examples/configuration/basic/hardware_fake.yaml +52 -0
  43. pytesdaqx-0.1.0/examples/configuration/basic/hardware_feb_signal_generator.yaml +86 -0
  44. pytesdaqx-0.1.0/examples/configuration/basic/run_setup.yaml +93 -0
  45. pytesdaqx-0.1.0/examples/configuration/calibration/acquisition_laser.yaml +33 -0
  46. pytesdaqx-0.1.0/examples/configuration/pxi_loopback/README.md +19 -0
  47. pytesdaqx-0.1.0/examples/configuration/pxi_loopback/acquisition_didv.yaml +43 -0
  48. pytesdaqx-0.1.0/examples/configuration/pxi_loopback/acquisition_iv_restore.yaml +56 -0
  49. pytesdaqx-0.1.0/examples/configuration/scans/acquisition_didv_ac.yaml +50 -0
  50. pytesdaqx-0.1.0/examples/configuration/scans/acquisition_didv_frequency_focus.yaml +49 -0
  51. pytesdaqx-0.1.0/examples/configuration/scans/acquisition_temperature_fake.yaml +53 -0
  52. pytesdaqx-0.1.0/examples/configuration/scans/acquisition_tes_bias_iv_didv.yaml +55 -0
  53. pytesdaqx-0.1.0/examples/configuration/scans/acquisition_transducer_vibration_scan.yaml +54 -0
  54. pytesdaqx-0.1.0/examples/configuration/temperature/hardware_lakeshore372.yaml +27 -0
  55. pytesdaqx-0.1.0/examples/configuration/temperature/hardware_macrt.yaml +59 -0
  56. pytesdaqx-0.1.0/examples/configuration/temperature/run_setup_lakeshore372.yaml +23 -0
  57. pytesdaqx-0.1.0/examples/configuration/temperature/run_setup_macrt.yaml +45 -0
  58. pytesdaqx-0.1.0/examples/simulation/README.md +9 -0
  59. pytesdaqx-0.1.0/examples/simulation/write_simulated_zarr.py +76 -0
  60. pytesdaqx-0.1.0/mkdocs.yml +59 -0
  61. pytesdaqx-0.1.0/pyproject.toml +95 -0
  62. pytesdaqx-0.1.0/pytesdaqx/__init__.py +10 -0
  63. pytesdaqx-0.1.0/pytesdaqx/acquisition/__init__.py +21 -0
  64. pytesdaqx-0.1.0/pytesdaqx/acquisition/fake.py +101 -0
  65. pytesdaqx-0.1.0/pytesdaqx/acquisition/nidaq_ai.py +723 -0
  66. pytesdaqx-0.1.0/pytesdaqx/acquisition/segment.py +39 -0
  67. pytesdaqx-0.1.0/pytesdaqx/acquisition/session.py +117 -0
  68. pytesdaqx-0.1.0/pytesdaqx/acquisition/streaming.py +167 -0
  69. pytesdaqx-0.1.0/pytesdaqx/analysis/__init__.py +3 -0
  70. pytesdaqx-0.1.0/pytesdaqx/analysis/continuity.py +244 -0
  71. pytesdaqx-0.1.0/pytesdaqx/cli/__init__.py +1 -0
  72. pytesdaqx-0.1.0/pytesdaqx/cli/build_manifest.py +149 -0
  73. pytesdaqx-0.1.0/pytesdaqx/cli/display_metadata.py +381 -0
  74. pytesdaqx-0.1.0/pytesdaqx/cli/laser.py +148 -0
  75. pytesdaqx-0.1.0/pytesdaqx/cli/run.py +184 -0
  76. pytesdaqx-0.1.0/pytesdaqx/cli/signal_generator.py +217 -0
  77. pytesdaqx-0.1.0/pytesdaqx/cli/squid.py +150 -0
  78. pytesdaqx-0.1.0/pytesdaqx/cli/temperature.py +127 -0
  79. pytesdaqx-0.1.0/pytesdaqx/cli/tes_bias.py +324 -0
  80. pytesdaqx-0.1.0/pytesdaqx/cli/transducer.py +73 -0
  81. pytesdaqx-0.1.0/pytesdaqx/cli/validate_config.py +48 -0
  82. pytesdaqx-0.1.0/pytesdaqx/cli/view_traces.py +964 -0
  83. pytesdaqx-0.1.0/pytesdaqx/config/__init__.py +30 -0
  84. pytesdaqx-0.1.0/pytesdaqx/config/durations.py +32 -0
  85. pytesdaqx-0.1.0/pytesdaqx/config/models.py +158 -0
  86. pytesdaqx-0.1.0/pytesdaqx/config/resolver.py +302 -0
  87. pytesdaqx-0.1.0/pytesdaqx/config/scan_values.py +141 -0
  88. pytesdaqx-0.1.0/pytesdaqx/config/schema.py +1052 -0
  89. pytesdaqx-0.1.0/pytesdaqx/config/temperature.py +419 -0
  90. pytesdaqx-0.1.0/pytesdaqx/config/validators.py +55 -0
  91. pytesdaqx-0.1.0/pytesdaqx/control/__init__.py +3 -0
  92. pytesdaqx-0.1.0/pytesdaqx/control/coordinator.py +1742 -0
  93. pytesdaqx-0.1.0/pytesdaqx/hardware/__init__.py +18 -0
  94. pytesdaqx-0.1.0/pytesdaqx/hardware/base.py +163 -0
  95. pytesdaqx-0.1.0/pytesdaqx/hardware/factory.py +48 -0
  96. pytesdaqx-0.1.0/pytesdaqx/hardware/legacy.py +157 -0
  97. pytesdaqx-0.1.0/pytesdaqx/hardware/registry.py +109 -0
  98. pytesdaqx-0.1.0/pytesdaqx/instruments/__init__.py +19 -0
  99. pytesdaqx-0.1.0/pytesdaqx/instruments/base.py +35 -0
  100. pytesdaqx-0.1.0/pytesdaqx/instruments/builtins.py +1632 -0
  101. pytesdaqx-0.1.0/pytesdaqx/instruments/communication.py +269 -0
  102. pytesdaqx-0.1.0/pytesdaqx/instruments/external.py +49 -0
  103. pytesdaqx-0.1.0/pytesdaqx/instruments/remote.py +443 -0
  104. pytesdaqx-0.1.0/pytesdaqx/instruments/signal_generator/__init__.py +0 -0
  105. pytesdaqx-0.1.0/pytesdaqx/instruments/signal_generator/agilent33500B.py +274 -0
  106. pytesdaqx-0.1.0/pytesdaqx/instruments/signal_generator/keithley2400.py +122 -0
  107. pytesdaqx-0.1.0/pytesdaqx/instruments/signal_generator/keysightDSOX1200.py +140 -0
  108. pytesdaqx-0.1.0/pytesdaqx/instruments/signal_generator/rigoldg800.py +253 -0
  109. pytesdaqx-0.1.0/pytesdaqx/instruments/temperature_controller/__init__.py +25 -0
  110. pytesdaqx-0.1.0/pytesdaqx/instruments/temperature_controller/base.py +40 -0
  111. pytesdaqx-0.1.0/pytesdaqx/instruments/temperature_controller/lakeshore.py +641 -0
  112. pytesdaqx-0.1.0/pytesdaqx/instruments/temperature_controller/macrt.py +195 -0
  113. pytesdaqx-0.1.0/pytesdaqx/instruments/temperature_controller/macrt_module.py +406 -0
  114. pytesdaqx-0.1.0/pytesdaqx/instruments/tes_squid/__init__.py +0 -0
  115. pytesdaqx-0.1.0/pytesdaqx/instruments/tes_squid/feb.py +657 -0
  116. pytesdaqx-0.1.0/pytesdaqx/instruments/tes_squid/magnicon.py +1340 -0
  117. pytesdaqx-0.1.0/pytesdaqx/io/__init__.py +69 -0
  118. pytesdaqx-0.1.0/pytesdaqx/io/acquisition_catalog.py +2205 -0
  119. pytesdaqx-0.1.0/pytesdaqx/io/catalog.py +17 -0
  120. pytesdaqx-0.1.0/pytesdaqx/io/hdf5_layout.py +93 -0
  121. pytesdaqx-0.1.0/pytesdaqx/io/hdf5_reader.py +405 -0
  122. pytesdaqx-0.1.0/pytesdaqx/io/metadata_compat.py +668 -0
  123. pytesdaqx-0.1.0/pytesdaqx/io/raw_data_reader.py +193 -0
  124. pytesdaqx-0.1.0/pytesdaqx/io/raw_data_writer.py +640 -0
  125. pytesdaqx-0.1.0/pytesdaqx/io/reader_utils.py +8 -0
  126. pytesdaqx-0.1.0/pytesdaqx/io/stream_reader.py +1372 -0
  127. pytesdaqx-0.1.0/pytesdaqx/io/zarr_reader.py +650 -0
  128. pytesdaqx-0.1.0/pytesdaqx/io/zarr_utils.py +550 -0
  129. pytesdaqx-0.1.0/pytesdaqx/naming.py +296 -0
  130. pytesdaqx-0.1.0/pytesdaqx/processing/__init__.py +3 -0
  131. pytesdaqx-0.1.0/pytesdaqx/processing/downsample.py +119 -0
  132. pytesdaqx-0.1.0/pytesdaqx/recorder/__init__.py +38 -0
  133. pytesdaqx-0.1.0/pytesdaqx/recorder/async_writer.py +84 -0
  134. pytesdaqx-0.1.0/pytesdaqx/recorder/base.py +31 -0
  135. pytesdaqx-0.1.0/pytesdaqx/recorder/hdf5.py +356 -0
  136. pytesdaqx-0.1.0/pytesdaqx/recorder/zarr.py +837 -0
  137. pytesdaqx-0.1.0/pytesdaqx/run/__init__.py +4 -0
  138. pytesdaqx-0.1.0/pytesdaqx/run/runner.py +107 -0
  139. pytesdaqx-0.1.0/pytesdaqx/run/yaml_run.py +2834 -0
  140. pytesdaqx-0.1.0/pytesdaqx/sequencer/__init__.py +10 -0
  141. pytesdaqx-0.1.0/pytesdaqx/sequencer/scan.py +98 -0
  142. pytesdaqx-0.1.0/pytesdaqx/sequencer/sequencer.py +117 -0
  143. pytesdaqx-0.1.0/pytesdaqx/services/__init__.py +24 -0
  144. pytesdaqx-0.1.0/pytesdaqx/services/laser.py +374 -0
  145. pytesdaqx-0.1.0/pytesdaqx/services/signal_generator_load.py +161 -0
  146. pytesdaqx-0.1.0/pytesdaqx/services/squid.py +489 -0
  147. pytesdaqx-0.1.0/pytesdaqx/services/temperature.py +472 -0
  148. pytesdaqx-0.1.0/pytesdaqx/services/tes_bias.py +1191 -0
  149. pytesdaqx-0.1.0/pytesdaqx/services/transducer.py +185 -0
  150. pytesdaqx-0.1.0/pytesdaqx/testing/__init__.py +0 -0
  151. pytesdaqx-0.1.0/pytesdaqx/testing/analyze_continuity.py +67 -0
  152. pytesdaqx-0.1.0/pytesdaqx/testing/lab_check.py +66 -0
  153. pytesdaqx-0.1.0/pytesdaqx/testing/ni_cli_common.py +82 -0
  154. pytesdaqx-0.1.0/pytesdaqx/testing/real_ni_benchmark.py +270 -0
  155. pytesdaqx-0.1.0/pytesdaqx/testing/real_ni_continuity_test.py +132 -0
  156. pytesdaqx-0.1.0/pytesdaqx/testing/real_ni_run_continuous.py +147 -0
  157. pytesdaqx-0.1.0/pytesdaqx/testing/real_ni_smoke.py +115 -0
  158. pytesdaqx-0.1.0/pytesdaqx/testing/real_ni_trigger_test.py +108 -0
  159. pytesdaqx-0.1.0/pytesdaqx.egg-info/PKG-INFO +93 -0
  160. pytesdaqx-0.1.0/pytesdaqx.egg-info/SOURCES.txt +209 -0
  161. pytesdaqx-0.1.0/pytesdaqx.egg-info/dependency_links.txt +1 -0
  162. pytesdaqx-0.1.0/pytesdaqx.egg-info/entry_points.txt +19 -0
  163. pytesdaqx-0.1.0/pytesdaqx.egg-info/requires.txt +30 -0
  164. pytesdaqx-0.1.0/pytesdaqx.egg-info/top_level.txt +1 -0
  165. pytesdaqx-0.1.0/setup.cfg +4 -0
  166. pytesdaqx-0.1.0/tests/test_acquisition_catalog_api_cleanup.py +119 -0
  167. pytesdaqx-0.1.0/tests/test_acquisition_catalog_naming.py +118 -0
  168. pytesdaqx-0.1.0/tests/test_api_cleanup.py +99 -0
  169. pytesdaqx-0.1.0/tests/test_cli_contract.py +78 -0
  170. pytesdaqx-0.1.0/tests/test_code_cleanup_regressions.py +66 -0
  171. pytesdaqx-0.1.0/tests/test_detector_settings_normalization.py +170 -0
  172. pytesdaqx-0.1.0/tests/test_developer_documentation.py +74 -0
  173. pytesdaqx-0.1.0/tests/test_distribution_readiness.py +74 -0
  174. pytesdaqx-0.1.0/tests/test_facility_catalog_metadata.py +156 -0
  175. pytesdaqx-0.1.0/tests/test_feb_driver.py +230 -0
  176. pytesdaqx-0.1.0/tests/test_feb_signal_generator_didv.py +227 -0
  177. pytesdaqx-0.1.0/tests/test_io_coordinate_cleanup.py +462 -0
  178. pytesdaqx-0.1.0/tests/test_lakeshore372_driver.py +319 -0
  179. pytesdaqx-0.1.0/tests/test_laser_configuration_ownership.py +210 -0
  180. pytesdaqx-0.1.0/tests/test_macrt_driver.py +231 -0
  181. pytesdaqx-0.1.0/tests/test_magnicon_driver.py +167 -0
  182. pytesdaqx-0.1.0/tests/test_manifest_and_zarr_metadata.py +172 -0
  183. pytesdaqx-0.1.0/tests/test_metadata_canonical_schema.py +543 -0
  184. pytesdaqx-0.1.0/tests/test_optional_hdf5_import.py +31 -0
  185. pytesdaqx-0.1.0/tests/test_partition_planning.py +226 -0
  186. pytesdaqx-0.1.0/tests/test_raw_data_writer.py +135 -0
  187. pytesdaqx-0.1.0/tests/test_release_cleanup.py +63 -0
  188. pytesdaqx-0.1.0/tests/test_repository_layout.py +112 -0
  189. pytesdaqx-0.1.0/tests/test_run_cli_cleanup.py +116 -0
  190. pytesdaqx-0.1.0/tests/test_scan_documentation.py +94 -0
  191. pytesdaqx-0.1.0/tests/test_scan_values.py +174 -0
  192. pytesdaqx-0.1.0/tests/test_signal_generator_cli.py +189 -0
  193. pytesdaqx-0.1.0/tests/test_signal_generator_drivers.py +389 -0
  194. pytesdaqx-0.1.0/tests/test_signal_generator_load_policy.py +300 -0
  195. pytesdaqx-0.1.0/tests/test_stream_reader_api_cleanup.py +218 -0
  196. pytesdaqx-0.1.0/tests/test_temperature_acquisition_integration.py +185 -0
  197. pytesdaqx-0.1.0/tests/test_temperature_cli.py +84 -0
  198. pytesdaqx-0.1.0/tests/test_temperature_configuration.py +285 -0
  199. pytesdaqx-0.1.0/tests/test_temperature_safety_cleanup.py +297 -0
  200. pytesdaqx-0.1.0/tests/test_temperature_service.py +114 -0
  201. pytesdaqx-0.1.0/tests/test_transducer_configuration_ownership.py +206 -0
  202. pytesdaqx-0.1.0/tests/test_user_documentation.py +104 -0
  203. pytesdaqx-0.1.0/tests/test_view_trace_partitions.py +432 -0
  204. pytesdaqx-0.1.0/tests/test_zarr_chunking_and_sharding.py +174 -0
  205. pytesdaqx-0.1.0/tests/test_zarr_compression.py +57 -0
  206. pytesdaqx-0.1.0/tests/test_zarr_default_and_naming.py +385 -0
  207. pytesdaqx-0.1.0/tests/test_zarr_manifest_sidecars.py +63 -0
  208. pytesdaqx-0.1.0/tests/test_zarr_metadata_layout.py +59 -0
  209. pytesdaqx-0.1.0/tests/test_zarr_native_backend.py +73 -0
  210. pytesdaqx-0.1.0/tests/test_zarr_read_block_terminology.py +166 -0
  211. pytesdaqx-0.1.0/tests/test_zarr_trace_labels.py +63 -0
@@ -0,0 +1,4 @@
1
+ include mkdocs.yml
2
+ recursive-include docs *.md
3
+ recursive-include examples *.md *.yaml *.py
4
+ prune site
@@ -0,0 +1,93 @@
1
+ Metadata-Version: 2.4
2
+ Name: pytesdaqx
3
+ Version: 0.1.0
4
+ Summary: TES detector acquisition, instrument control, and raw-data recording
5
+ Author: TESSERACT DAQ developers
6
+ Classifier: Development Status :: 3 - Alpha
7
+ Classifier: Intended Audience :: Science/Research
8
+ Classifier: Operating System :: POSIX :: Linux
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: Programming Language :: Python :: 3.11
11
+ Classifier: Programming Language :: Python :: 3.12
12
+ Classifier: Topic :: Scientific/Engineering :: Physics
13
+ Requires-Python: <3.13,>=3.11
14
+ Description-Content-Type: text/markdown
15
+ Requires-Dist: numpy<2.0,>=1.26
16
+ Requires-Dist: PyYAML
17
+ Requires-Dist: h5py>=3.10
18
+ Requires-Dist: matplotlib
19
+ Requires-Dist: scipy
20
+ Requires-Dist: zarr<4,>=3
21
+ Requires-Dist: numcodecs
22
+ Requires-Dist: zstandard
23
+ Provides-Extra: dev
24
+ Requires-Dist: pytest; extra == "dev"
25
+ Requires-Dist: ruff; extra == "dev"
26
+ Requires-Dist: build; extra == "dev"
27
+ Requires-Dist: twine; extra == "dev"
28
+ Provides-Extra: ni
29
+ Requires-Dist: nidaqmx; extra == "ni"
30
+ Provides-Extra: visa
31
+ Requires-Dist: pyvisa; extra == "visa"
32
+ Provides-Extra: magnicon
33
+ Requires-Dist: paramiko; extra == "magnicon"
34
+ Provides-Extra: notebook
35
+ Requires-Dist: jupyter; extra == "notebook"
36
+ Provides-Extra: docs
37
+ Requires-Dist: mkdocs<2,>=1.6; extra == "docs"
38
+ Requires-Dist: mkdocs-material<10,>=9.5; extra == "docs"
39
+
40
+ # pytesdaqx
41
+
42
+ `pytesdaqx` is the Python acquisition and instrument-control framework used for TES detector readout. It supports continuous and finite NI-DAQmx acquisition, TES and SQUID control, YAML-defined sequences, and raw-data recording in Zarr or legacy HDF5. Zarr is the default for new
43
+ acquisition configuration files when `run.file_format` is omitted. **Supported Python versions
44
+ are 3.11 and 3.12.** Python 3.10 is not supported: the required `zarr>=3` has no published
45
+ wheel/sdist for 3.10.
46
+
47
+ ## Documentation
48
+
49
+ The user manual is in [`docs/`](docs/index.md). Start with:
50
+
51
+ - [Installation](docs/installation/index.md)
52
+ - [Quick start](docs/user/quickstart.md)
53
+ - [Configuration files](docs/user/configuration/index.md)
54
+ - [Command-line tools](docs/user/cli/index.md)
55
+ - [Terminology](docs/user/terminology.md)
56
+ - [Data format](docs/user/data_format.md)
57
+ - [Developer guide](docs/developer/index.md)
58
+
59
+ To build the documentation website locally:
60
+
61
+ ```bash
62
+ python -m pip install -e '.[docs]'
63
+ mkdocs serve
64
+ ```
65
+
66
+ ## Minimal local test
67
+
68
+ ```bash
69
+ python -m pip install -e '.[dev]'
70
+
71
+ pytesdaqx-validate-config \
72
+ --hardware examples/configuration/basic/hardware_fake.yaml \
73
+ --run-setup examples/configuration/basic/run_setup.yaml \
74
+ --acquisition-config examples/configuration/basic/acquisition_background.yaml
75
+
76
+ pytesdaqx-run \
77
+ --hardware examples/configuration/basic/hardware_fake.yaml \
78
+ --run-setup examples/configuration/basic/run_setup.yaml \
79
+ --acquisition-config examples/configuration/basic/acquisition_background.yaml \
80
+ --backend fake
81
+ ```
82
+
83
+ The fake backend exercises configuration resolution and acquisition orchestration without contacting hardware.
84
+
85
+ ## Development
86
+
87
+ Run the repository tests with:
88
+
89
+ ```bash
90
+ pytest -q
91
+ ```
92
+
93
+ Architecture, extension points, hardware support, testing, and release practices are documented in the [developer guide](docs/developer/index.md). The maintained example configurations are described in [`examples/README.md`](examples/README.md).
@@ -0,0 +1,54 @@
1
+ # pytesdaqx
2
+
3
+ `pytesdaqx` is the Python acquisition and instrument-control framework used for TES detector readout. It supports continuous and finite NI-DAQmx acquisition, TES and SQUID control, YAML-defined sequences, and raw-data recording in Zarr or legacy HDF5. Zarr is the default for new
4
+ acquisition configuration files when `run.file_format` is omitted. **Supported Python versions
5
+ are 3.11 and 3.12.** Python 3.10 is not supported: the required `zarr>=3` has no published
6
+ wheel/sdist for 3.10.
7
+
8
+ ## Documentation
9
+
10
+ The user manual is in [`docs/`](docs/index.md). Start with:
11
+
12
+ - [Installation](docs/installation/index.md)
13
+ - [Quick start](docs/user/quickstart.md)
14
+ - [Configuration files](docs/user/configuration/index.md)
15
+ - [Command-line tools](docs/user/cli/index.md)
16
+ - [Terminology](docs/user/terminology.md)
17
+ - [Data format](docs/user/data_format.md)
18
+ - [Developer guide](docs/developer/index.md)
19
+
20
+ To build the documentation website locally:
21
+
22
+ ```bash
23
+ python -m pip install -e '.[docs]'
24
+ mkdocs serve
25
+ ```
26
+
27
+ ## Minimal local test
28
+
29
+ ```bash
30
+ python -m pip install -e '.[dev]'
31
+
32
+ pytesdaqx-validate-config \
33
+ --hardware examples/configuration/basic/hardware_fake.yaml \
34
+ --run-setup examples/configuration/basic/run_setup.yaml \
35
+ --acquisition-config examples/configuration/basic/acquisition_background.yaml
36
+
37
+ pytesdaqx-run \
38
+ --hardware examples/configuration/basic/hardware_fake.yaml \
39
+ --run-setup examples/configuration/basic/run_setup.yaml \
40
+ --acquisition-config examples/configuration/basic/acquisition_background.yaml \
41
+ --backend fake
42
+ ```
43
+
44
+ The fake backend exercises configuration resolution and acquisition orchestration without contacting hardware.
45
+
46
+ ## Development
47
+
48
+ Run the repository tests with:
49
+
50
+ ```bash
51
+ pytest -q
52
+ ```
53
+
54
+ Architecture, extension points, hardware support, testing, and release practices are documented in the [developer guide](docs/developer/index.md). The maintained example configurations are described in [`examples/README.md`](examples/README.md).
@@ -0,0 +1,118 @@
1
+ # Acquisition pipeline
2
+
3
+ ## Backend contract
4
+
5
+ There is not yet a formal abstract base class, but acquisition backends follow this minimal protocol:
6
+
7
+ ```python
8
+ backend.configure(
9
+ measurement_name=...,
10
+ measurement_config=...,
11
+ detector_channels=...,
12
+ )
13
+ segment = backend.read_segment()
14
+ backend.close()
15
+ ```
16
+
17
+ `read_segment()` returns a `Segment` with a two-dimensional array shaped `[channel, sample]`.
18
+
19
+ ## DAQSession
20
+
21
+ `DAQSession` owns the selected measurement dictionary and backend. For every configuration call it:
22
+
23
+ 1. copies the measurement configuration;
24
+ 2. deep-merges temporary overrides;
25
+ 3. selects detector channels;
26
+ 4. calls `backend.configure()`.
27
+
28
+ This copy is important: scan- or step-specific overrides must not mutate the original resolved measurement dictionary.
29
+
30
+ ## Run-controller acquisition modes
31
+
32
+ `YAMLRunController._run_step()` chooses one of three paths.
33
+
34
+ ### Threaded continuous
35
+
36
+ Used when:
37
+
38
+ ```text
39
+ adc.mode == continuous
40
+ and threaded_continuous is enabled
41
+ ```
42
+
43
+ ```text
44
+ NIDAQAIBackend.read_segment()
45
+ -> bounded SegmentRingBuffer
46
+ -> consumer callback
47
+ -> processors/callbacks/recorder
48
+ ```
49
+
50
+ The queue blocks by default when full. `drop_when_full: true` is suitable only when losing blocks is acceptable, such as some live-display workflows. Raw-data acquisition should normally preserve the blocking behavior.
51
+
52
+ ### Serial or serial rearm
53
+
54
+ Finite or triggered measurements use the serial path. If the measurement is finite or has an explicit trigger, the backend is configured and closed for every trace so the NI task rearms cleanly.
55
+
56
+ A non-threaded continuous-mode measurement can also use the serial path without rearming between blocks.
57
+
58
+ ### PXI phase-locked dIdV
59
+
60
+ A dIdV measurement using PXIe-4322 AC without an explicit trigger can use the dedicated phase-locked stream path:
61
+
62
+ 1. Configure and arm AI.
63
+ 2. Configure AI to wait on the AO StartTrigger.
64
+ 3. Start the multi-channel PXI AO waveform task.
65
+ 4. Read contiguous integer-period records from one AI task.
66
+ 5. Optionally discard initial transition records.
67
+ 6. Keep the AI sample clock referenced to the configured PXI reference clock.
68
+
69
+ Explicit digital or analog triggers retain the finite-rearm diagnostic path.
70
+
71
+ ## NI analog-input backend
72
+
73
+ `NIDAQAIBackend` currently supports:
74
+
75
+ - one single DAQmx task for the selected AI channels;
76
+ - continuous and finite sample modes;
77
+ - digital-edge/external start triggers;
78
+ - analog-edge/threshold start triggers;
79
+ - scaled floating-point reads;
80
+ - unscaled `int16` reads with device calibration coefficients;
81
+ - optional input-buffer and read-chunk sizing;
82
+ - optional reference-clock configuration;
83
+ - logical `pfi0` resolution when selected channels use one ADC device.
84
+
85
+ Selected channels must fit in one DAQmx task. Explicit multi-device synchronized AI is not implemented in this backend.
86
+
87
+ For long continuous saved blocks, the backend may perform several shorter DAQmx reads into temporary contiguous arrays. This limits DAQmx buffer allocation while preserving the requested saved write-block length.
88
+
89
+ ## Processing and callbacks
90
+
91
+ Every acquired `Segment` passes through:
92
+
93
+ ```text
94
+ processor 1 -> processor 2 -> ... -> callback 1 -> callback 2 -> recorder
95
+ ```
96
+
97
+ Processors may replace the `Segment`; callbacks receive the processed result. The current built-in processor is `DownsampleProcessor`.
98
+
99
+ `collect_records=True` retains records in memory and should be limited to tests, notebooks, and short runs.
100
+
101
+ ## Metadata propagation
102
+
103
+ Before recording, the run controller adds:
104
+
105
+ - measurement label;
106
+ - sequence-step index;
107
+ - compact stream-level scan metadata;
108
+ - an opt-in stream-level temperature snapshot;
109
+ - backend metadata such as DAQmx read mode;
110
+ - ADC conversion coefficients when available.
111
+
112
+ The temperature snapshot is taken once per stream after scan and sequence controls are applied. It contains only sensor device/channel, temperature, resistance, and units; read errors are nonfatal and represented compactly. Recorders translate nested dictionaries to native Zarr attributes or JSON-encoded HDF5 attributes. Readers normalize both forms back to dictionaries.
113
+
114
+ ## Resource lifecycle
115
+
116
+ A backend must make `close()` idempotent. Configuration failures must close partially created tasks. Instrument outputs that can remain active require explicit service/coordinator cleanup semantics; do not rely on object destruction.
117
+
118
+ Temperature scan cleanup is executed by the outer run-controller `finally` block before `HardwareRegistry.close_all()`, including on Ctrl-C. The narrow shutdown path is `set_setpoint(0)` followed by `disable_heater()`; it does not call general PID/topology configuration. If the run already has an exception, cleanup or close failures are logged while the original exception is preserved.
@@ -0,0 +1,113 @@
1
+ # Adding hardware
2
+
3
+ Choose the smallest extension point that matches the device.
4
+
5
+ ## 1. New acquisition backend
6
+
7
+ Use this path for a device that produces sampled detector data.
8
+
9
+ Implement the backend protocol:
10
+
11
+ ```python
12
+ class MyBackend:
13
+ def configure(self, *, measurement_name, measurement_config, detector_channels): ...
14
+ def read_segment(self) -> Segment: ...
15
+ def close(self) -> None: ...
16
+ ```
17
+
18
+ Requirements:
19
+
20
+ - Return data shaped `[channel, sample]`.
21
+ - Preserve the requested detector-channel order.
22
+ - Include sample rate and useful backend metadata.
23
+ - Make `close()` safe after partial configuration and safe to call more than once.
24
+ - Avoid importing vendor libraries at module import time when possible.
25
+
26
+ Then add backend construction in `pytesdaqx.run.yaml_run._make_backend()` and expose a CLI choice if it is user-selectable. Add fake/unit tests and a real-hardware diagnostic command.
27
+
28
+ ## 2. New built-in controllable instrument
29
+
30
+ Use this path for a device that sets bias, waveform, SQUID, laser, or temperature state.
31
+
32
+ 1. Add a wrapper in `pytesdaqx.instruments`.
33
+ 2. Prefer lazy vendor-driver construction in `open()`.
34
+ 3. Implement generic capability methods used by the relevant service.
35
+ 4. Add the canonical driver name to `BUILTIN_DRIVER_NAMES`.
36
+ 5. Add a branch in `build_builtin_instrument()`.
37
+ 6. Add hardware YAML documentation and tests using a fake underlying instance.
38
+ 7. Add the path to the hardware support matrix after validation.
39
+
40
+ Keep communication details in the instrument wrapper, not in services or sequence code.
41
+
42
+ ## 3. Out-of-tree/private instrument
43
+
44
+ For site-specific code, use:
45
+
46
+ ```yaml
47
+ instruments:
48
+ private_device:
49
+ driver: external
50
+ class_path: package.module:DeviceClass
51
+ constructor_args: []
52
+ constructor_kwargs: {}
53
+ connection:
54
+ address: ...
55
+ ```
56
+
57
+ `ExternalInstrument` imports the class lazily and delegates unknown methods to the instance. The external class still needs to expose methods expected by the selected service.
58
+
59
+ This route is preferable when a driver cannot be distributed with `pytesdaqx`.
60
+
61
+ ## 4. Custom factory registration
62
+
63
+ Applications embedding `pytesdaqx` can register a builder:
64
+
65
+ ```python
66
+ factory = DriverFactory.with_default_fake_builders(allow_fake=False)
67
+ factory.register("my_driver", lambda name, cfg: MyInstrument(name=name, config=cfg))
68
+ registry = HardwareRegistry(hardware_config, factory=factory)
69
+ ```
70
+
71
+ Use `allow_fake=False` in production validation when an unknown driver must be an error rather than silently becoming `FakeHardwareDevice`.
72
+
73
+ ## 5. New service path
74
+
75
+ Add a service path only when the same logical operation needs genuinely different behavior, conversion, grouping, or state handling.
76
+
77
+ Examples already present in `TESBiasService` are PXI AO, direct signal generator, and FEB-plus-signal-generator paths.
78
+
79
+ A new path should define:
80
+
81
+ - accepted physical units and conversion rules;
82
+ - set and readback behavior;
83
+ - cached-state policy when readback is absent;
84
+ - grouped-channel behavior if synchronization matters;
85
+ - startup and cleanup behavior;
86
+ - metadata state source and error reporting.
87
+
88
+ ## 6. New scan/control target
89
+
90
+ A new scan target is a cross-cutting feature. Update:
91
+
92
+ 1. scan/config schema validation;
93
+ 2. `ControlCoordinator` target dispatch;
94
+ 3. channel-selection logic;
95
+ 4. service call and unit conversion;
96
+ 5. settling behavior;
97
+ 6. metadata persistence;
98
+ 7. fake and target-specific tests;
99
+ 8. user scan documentation.
100
+
101
+ ## Required tests
102
+
103
+ At minimum, test:
104
+
105
+ - hardware YAML resolution;
106
+ - registry construction;
107
+ - service path selection;
108
+ - the generic set/get methods;
109
+ - failure cleanup;
110
+ - a fake full-stack measurement or scan;
111
+ - import behavior without the vendor library installed.
112
+
113
+ Real-hardware validation should record the OS, kernel, driver version, model, physical wiring, command, and observed result.
@@ -0,0 +1,142 @@
1
+ # Architecture
2
+
3
+ ## End-to-end flow
4
+
5
+ A normal `pytesdaqx-run` invocation follows this path:
6
+
7
+ ```text
8
+ pytesdaqx-run
9
+ |
10
+ v
11
+ YAMLRunController.from_yaml()
12
+ |
13
+ +--> ConfigResolver
14
+ | hardware.yaml + run_setup.yaml + acquisition.yaml
15
+ | -> ResolvedConfig
16
+ |
17
+ +--> HardwareRegistry
18
+ | -> controllable instrument objects
19
+ |
20
+ +--> detector-oriented services
21
+ | TESBiasService
22
+ | SQUIDService
23
+ | LaserService
24
+ | TemperatureService
25
+ |
26
+ +--> acquisition backend
27
+ | NIDAQAIBackend or FakeAcquisitionBackend
28
+ |
29
+ v
30
+ YAMLRunController.run()
31
+ |
32
+ +--> ScanEngine: build scan points
33
+ +--> ControlCoordinator: apply controls and manage measurement state
34
+ +--> DAQSession: configure the selected measurement
35
+ +--> acquisition loop: produce Segment objects
36
+ +--> optional processors and callbacks
37
+ +--> ZarrRecorder or HDF5Recorder
38
+ +--> acquisition manifest
39
+ ```
40
+
41
+ The run controller owns orchestration. It should not contain low-level instrument protocol details, and a recorder should never operate hardware.
42
+
43
+ ## Control plane and data plane
44
+
45
+ It is useful to view the system as two connected planes.
46
+
47
+ ### Control plane
48
+
49
+ ```text
50
+ ResolvedConfig
51
+ -> HardwareRegistry
52
+ -> services
53
+ -> ControlCoordinator
54
+ -> scan point / sequence-step actions
55
+ ```
56
+
57
+ The control plane sets TES bias, SQUID state, laser output, and temperature setpoints. It also performs cleanup after every sequence step, including failed steps.
58
+
59
+ ### Data plane
60
+
61
+ ```text
62
+ ResolvedConfig.channel_map()
63
+ -> acquisition backend
64
+ -> Segment(data[channel, sample])
65
+ -> processor(s)
66
+ -> callback(s)
67
+ -> recorder
68
+ -> raw-data reader
69
+ ```
70
+
71
+ The data plane uses detector-channel order from the resolved configuration. Storage metadata preserves that order and the physical ADC mapping.
72
+
73
+ ## Why services exist
74
+
75
+ Run setup is detector-oriented: a user asks to set the TES bias for `TES_A`, not to invoke a particular driver method on AO channel 0. Services convert that detector-oriented request into a device-specific path.
76
+
77
+ For example, `TESBiasService` can construct one of three paths:
78
+
79
+ ```text
80
+ same device, PXI-like AO
81
+ -> PXI4322BiasPath
82
+
83
+ same device, signal-generator-like
84
+ -> SignalGeneratorBiasPath
85
+
86
+ FEB for DC + signal generator for AC
87
+ -> FEBPlusSignalGeneratorBiasPath
88
+ ```
89
+
90
+ The split FEB path has different waveform-center semantics from the other two:
91
+ the FEB carries TES DC, while the signal generator remains zero-centered and
92
+ provides only the AC perturbation. If several detector channels share one
93
+ generator output, the service resolves their physical waveform settings first,
94
+ requires them to match, configures the generator once, and then connects the
95
+ individual FEB inputs.
96
+
97
+ This keeps sequence and scan code independent of the physical implementation.
98
+
99
+ ## NI ADC ownership
100
+
101
+ The NI analog-input device is intentionally different from controllable instruments:
102
+
103
+ - `ConfigResolver` produces the ADC channel map.
104
+ - `NIDAQAIBackend` creates and owns the NI-DAQmx AI task.
105
+ - The default `HardwareRegistry` does **not** create the ADC task.
106
+
107
+ A `hardware.yaml` entry with `driver: nidaqmx` is therefore used for inventory and channel resolution. The actual AI task is created by `NIDAQAIBackend` from `ResolvedConfig.channel_map()`.
108
+
109
+ ## Run hierarchy
110
+
111
+ The main orchestration hierarchy is:
112
+
113
+ ```text
114
+ acquisition
115
+ -> sequence repeat
116
+ -> scan point
117
+ -> sequence step / stream
118
+ -> acquired in-memory record
119
+ ```
120
+
121
+ For each sequence step, the run controller opens one stream recorder, executes the acquisition path, and closes the stream recorder. Zarr therefore has one stream group per executed step. Repeated steps create distinct full stream names. Stream IDs are the bare `I#_D#_T#` timestamp identifiers.
122
+
123
+ ## Internal `Segment` name
124
+
125
+ The in-memory acquisition object is currently named `Segment` for historical reasons. It is always a two-dimensional array:
126
+
127
+ ```text
128
+ data[channel, sample]
129
+ ```
130
+
131
+ Its external meaning depends on the measurement and recorder:
132
+
133
+ - Continuous acquisition: one internal acquisition read block appended to a continuous stream.
134
+ - Finite or triggered acquisition: one trace.
135
+
136
+ Do not expose the internal `Segment` class name as Zarr storage terminology. In user-facing documentation, use **read block**, **trace**, or **partition** as appropriate.
137
+
138
+ ## Failure and cleanup model
139
+
140
+ The run controller uses nested `try/finally` blocks around measurement setup, acquisition, recorder closure, and hardware cleanup. This is especially important for NI AO tasks: a partially configured PXI waveform task must be closed so the next run does not fail because the NI resource remains reserved.
141
+
142
+ Manifest updates are intentionally rebuildable. A manifest callback failure is logged but must not corrupt or stop raw-data writing.
@@ -0,0 +1,136 @@
1
+ # Configuration pipeline
2
+
3
+ The user edits three YAML files, but runtime code consumes one `ResolvedConfig`.
4
+
5
+ ## Resolution stages
6
+
7
+ ```text
8
+ hardware.yaml
9
+ run_setup.yaml
10
+ acquisition.yaml
11
+ |
12
+ v
13
+ ConfigResolver._load_yaml()
14
+ |
15
+ v
16
+ hardware entries -> HardwareDevice
17
+ |
18
+ v
19
+ schema validation and cross-reference checks
20
+ |
21
+ v
22
+ selected detector channels -> ResolvedChannel[]
23
+ |
24
+ v
25
+ logical temperatures -> ResolvedTemperaturePath[]
26
+ |
27
+ v
28
+ measurements + sequence -> SequenceStep[]
29
+ |
30
+ v
31
+ scan -> ScanParameter[]
32
+ |
33
+ v
34
+ ResolvedConfig
35
+ ```
36
+
37
+ `ConfigResolver.resolve()` is the canonical entry point.
38
+
39
+ ## Resolved channel order
40
+
41
+ Detector channels are sorted by:
42
+
43
+ ```text
44
+ ADC device name, ADC channel number, detector-channel name
45
+ ```
46
+
47
+ The resulting `array_index` defines row order in acquired arrays and recorded channel maps. Code must not assume that YAML insertion order is the storage order.
48
+
49
+ ## Validation layers
50
+
51
+ Configuration checking has several layers:
52
+
53
+ 1. YAML must parse to a top-level mapping.
54
+ 2. `validate_full_configuration()` checks schemas and cross-file relationships.
55
+ 3. The resolver verifies referenced devices and physical channels.
56
+ 4. Duplicate ADC assignments are rejected.
57
+ 5. Sequence and scan blocks are normalized into dataclasses.
58
+ 6. In strict mode, active channels must be part of the available detector-channel set.
59
+ 7. Backends and services perform hardware-specific validation when configured.
60
+
61
+ The validator can return warnings in `ResolvedConfig.warnings`; callers should display or log them rather than silently discard them.
62
+
63
+ ## Precedence
64
+
65
+ The main precedence rule is:
66
+
67
+ ```text
68
+ explicit Python/CLI argument
69
+ > acquisition-config YAML run option
70
+ > internal default
71
+ ```
72
+
73
+ Channel selection follows:
74
+
75
+ ```text
76
+ CLI channels
77
+ > run.channels
78
+ > every channel in run_setup.yaml
79
+ ```
80
+
81
+ Measurement-specific temporary overrides are deep-merged by `DAQSession.configure_measurement()` before the backend is configured. This is used, for example, to resolve `read_block_duration`, `trace_duration`, or exact period counts into a backend sample count.
82
+
83
+ See the [user precedence guide](../user/configuration/resolution.md) for the user-facing rules.
84
+
85
+ ## Temperature-path resolution
86
+
87
+ `resolve_temperature_paths()` converts canonical `run_setup.temperatures` entries into `ResolvedTemperaturePath` objects. Each path contains one required sensor endpoint and one optional control endpoint. Endpoint references are checked against the controller's explicit `sensors` or `heaters` inventory, including MACRT module aliases. Singular/flat temperature syntax and endpoint `controller` aliases are rejected.
88
+
89
+ A control endpoint also carries its heater-level `setpoint_limits` normalized to kelvin. The service layer attaches registry device instances and enforces those limits centrally before calling a driver, so CLI, controls, scans, and cleanup have identical safety behavior. Drivers implement one fixed Kelvin-based interface (`read_temperature_k`, `read_resistance_ohm`, `read_setpoint_k`, `set_setpoint_k`, `get_control_state`, and `disable_heater`) instead of relying on legacy method-name alternatives.
90
+
91
+ ## Laser configuration ownership
92
+
93
+ Laser configuration is intentionally split across two layers:
94
+
95
+ - `run_setup.yaml` owns stable topology: logical laser name, physical device, control channel, and TTL/reference channel.
96
+ - the acquisition configuration owns operational pulse settings: voltage levels, offset, pulse width, repetition frequency, waveform shape, phase, and output cleanup behavior.
97
+
98
+ `LaserService` is constructed from run setup and therefore owns the physical output channels. `ControlCoordinator` applies measurement-level pulse settings without passing a channel override. Metadata assembly combines the run-setup topology with the acquisition pulse settings so the stored stream describes both what was wired and what was requested. Schema validation rejects operational fields in run setup and device/channel overrides in acquisition laser sections.
99
+
100
+ ## Scan representation
101
+
102
+ The resolver produces `ScanParameter` objects. `ScanEngine` converts them into ordered `ScanPoint` values using either Cartesian product or zipped axes. A temperature parameter may also carry normalized `wait_for_stable` configuration. The run controller applies each scan point, performs any required temperature-stability wait, then executes the full sequence. Only `temperature.setpoint` is accepted for temperature scans.
103
+
104
+ `scan.cleanup.temperature` remains raw acquisition configuration because it runs once after the whole scan rather than generating scan points. `YAMLRunController` invokes it from the outer run `finally` block before closing the registry. The coordinator sets an optional cleanup setpoint first, then calls the narrow service `disable_heater()` method and verifies the reported off state when requested. Original acquisition exceptions take precedence over cleanup errors.
105
+
106
+ Developers adding a scan target normally need to update all of:
107
+
108
+ - `config/schema.py` validation;
109
+ - `ControlCoordinator.apply_target()` or a target-specific helper;
110
+ - metadata capture when the value must be persisted;
111
+ - user scan documentation;
112
+ - fake-path and target-specific tests.
113
+
114
+ ## Adding a configuration field
115
+
116
+ Before adding a YAML key, decide which domain owns it:
117
+
118
+ - Inventory or connection property: `hardware.yaml`.
119
+ - Detector wiring or calibration constant: `run_setup.yaml`.
120
+ - Acquisition behavior or sequence policy: `acquisition.yaml`.
121
+
122
+ Then update, as applicable:
123
+
124
+ 1. schema validation;
125
+ 2. resolver normalization or dataclasses;
126
+ 3. the consumer backend/service/recorder;
127
+ 4. resolved metadata snapshots;
128
+ 5. an example YAML file;
129
+ 6. user documentation;
130
+ 7. unit and regression tests.
131
+
132
+ Prefer one canonical name. Compatibility aliases should be translated at the configuration boundary and clearly documented as aliases rather than propagated throughout the code.
133
+
134
+ ## Source and resolved configuration in data
135
+
136
+ For Zarr acquisitions, the top-level acquisition attributes contain source YAML snapshots and a resolved summary. HDF5 acquisitions use a `config/` directory beside the data files. This metadata is part of reproducibility; changes to resolution behavior should be considered data-format changes even when array layout is unchanged.