tdmspy 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- tdmspy-0.1.0/.gitattributes +3 -0
- tdmspy-0.1.0/.github/dependabot.yml +6 -0
- tdmspy-0.1.0/.github/workflows/publish.yaml +59 -0
- tdmspy-0.1.0/.github/workflows/test.yaml +47 -0
- tdmspy-0.1.0/.gitignore +19 -0
- tdmspy-0.1.0/.python-version +1 -0
- tdmspy-0.1.0/LICENSE +21 -0
- tdmspy-0.1.0/PKG-INFO +76 -0
- tdmspy-0.1.0/README.md +66 -0
- tdmspy-0.1.0/doc/design.md +152 -0
- tdmspy-0.1.0/doc/special-cases.md +146 -0
- tdmspy-0.1.0/justfile +67 -0
- tdmspy-0.1.0/pyproject.toml +32 -0
- tdmspy-0.1.0/tdmspy/__init__.py +31 -0
- tdmspy-0.1.0/tdmspy/_format.py +61 -0
- tdmspy-0.1.0/tdmspy/_metadata.py +205 -0
- tdmspy-0.1.0/tdmspy/_scaling.py +69 -0
- tdmspy-0.1.0/tdmspy/_scan.py +269 -0
- tdmspy-0.1.0/tdmspy/errors.py +17 -0
- tdmspy-0.1.0/tdmspy/file.py +236 -0
- tdmspy-0.1.0/tdmspy/py.typed +0 -0
- tdmspy-0.1.0/tdmspy/timestamp.py +57 -0
- tdmspy-0.1.0/tests/__init__.py +0 -0
- tdmspy-0.1.0/tests/builder.py +328 -0
- tdmspy-0.1.0/tests/catalogue.py +112 -0
- tdmspy-0.1.0/tests/conftest.py +35 -0
- tdmspy-0.1.0/tests/data/real/expected.json +553 -0
- tdmspy-0.1.0/tests/data/real/one_int16_channel.tdms +0 -0
- tdmspy-0.1.0/tests/data/real/three_int16_channels.tdms +0 -0
- tdmspy-0.1.0/tests/data/real/three_int16_channels_short.tdms +0 -0
- tdmspy-0.1.0/tests/data/real/unequal_counts_untyped_channels.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/big_endian.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/big_endian_interleaved.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/booleans.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/channel_without_data.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/chunks.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/complex_numbers.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/contiguous_three_types.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/daqmx_raw_data.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/empty_file.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/extended_float_channel.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/first_appearance_order.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/floats.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/garbage_after_segments.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/group_without_channels.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/huge_object_count.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/huge_path_length.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/huge_property_count.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/huge_string_length.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/huge_value_count.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/implicit_group.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/index_file.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/index_none_then_data.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/index_same_after_change.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/index_same_without_previous.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/index_type_change.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/integers.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/interleaved_chunks.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/interleaved_mixed_types.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/interleaved_three_channels_floats.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/interleaved_two_channels.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/interleaved_unequal_counts.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/many_properties.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/many_segments.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/meta_without_new_list.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/metadata_only_segment.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/minimal.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/mixed_endianness.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/new_list_replaces.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/next_offset_beyond_end.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/next_offset_unknown.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/path_without_slash.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/property_complex.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/property_extended_float.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/property_odd_values.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/property_types.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/property_unicode.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/property_unit_types.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/property_update.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/quotes_and_slashes.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/raw_offset_padding.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/raw_only_segment.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/reserved_looking_names.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/root_only.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/scale_already_scaled.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/scale_chain.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/scale_channel_overrides_group.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/scale_file_level.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/scale_float32_input.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/scale_group_level.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/scale_linear.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/scale_none.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/scale_polynomial.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/scale_unsupported_type.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/shorter_than_lead_in.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/strings.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/three_level_path.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/timestamp_fraction_extremes.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/timestamps.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/truncated_after_whole_chunks.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/truncated_in_lead_in.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/truncated_in_metadata.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/truncated_interleaved_row.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/truncated_mid_values.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/truncated_second_channel.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/unequal_counts.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/version_4712.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/version_unknown.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/wrong_signature.tdms +0 -0
- tdmspy-0.1.0/tests/data/synthetic/zero_values.tdms +0 -0
- tdmspy-0.1.0/tests/gen_catalogue_doc.py +77 -0
- tdmspy-0.1.0/tests/gen_real_expected.py +78 -0
- tdmspy-0.1.0/tests/test_builder.py +93 -0
- tdmspy-0.1.0/tests/test_catalogue_coverage.py +49 -0
- tdmspy-0.1.0/tests/test_fuzz.py +103 -0
- tdmspy-0.1.0/tests/test_privacy.py +114 -0
- tdmspy-0.1.0/tests/test_reader_api.py +157 -0
- tdmspy-0.1.0/tests/test_reader_testcases.py +44 -0
- tdmspy-0.1.0/tests/test_real_files.py +51 -0
- tdmspy-0.1.0/tests/test_robust.py +57 -0
- tdmspy-0.1.0/tests/test_synthetic_files.py +24 -0
- tdmspy-0.1.0/tests/test_timestamp.py +50 -0
- tdmspy-0.1.0/tests/testcases.py +1049 -0
- tdmspy-0.1.0/tests/testcases_cli.py +35 -0
- tdmspy-0.1.0/tests/truth.py +231 -0
- tdmspy-0.1.0/uv.lock +174 -0
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# Source: https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/
|
|
2
|
+
|
|
3
|
+
name: Build Python Package
|
|
4
|
+
|
|
5
|
+
on: push
|
|
6
|
+
|
|
7
|
+
jobs:
|
|
8
|
+
build:
|
|
9
|
+
name: Build package
|
|
10
|
+
runs-on: ubuntu-latest
|
|
11
|
+
|
|
12
|
+
steps:
|
|
13
|
+
- uses: actions/checkout@v7
|
|
14
|
+
with:
|
|
15
|
+
persist-credentials: false
|
|
16
|
+
|
|
17
|
+
- name: Set up Python
|
|
18
|
+
uses: actions/setup-python@v7
|
|
19
|
+
with:
|
|
20
|
+
python-version: "3.13"
|
|
21
|
+
|
|
22
|
+
- name: Install build package
|
|
23
|
+
run: >
|
|
24
|
+
python3 -m
|
|
25
|
+
pip install
|
|
26
|
+
build
|
|
27
|
+
--user
|
|
28
|
+
|
|
29
|
+
- name: Build a binary wheel and a source tarball
|
|
30
|
+
run: python3 -m build
|
|
31
|
+
|
|
32
|
+
- name: Store the distribution packages
|
|
33
|
+
uses: actions/upload-artifact@v7
|
|
34
|
+
with:
|
|
35
|
+
name: python-package-distributions
|
|
36
|
+
path: dist/
|
|
37
|
+
|
|
38
|
+
publish:
|
|
39
|
+
name: Publish package
|
|
40
|
+
# Only publish to PyPI on tag pushes
|
|
41
|
+
if: startsWith(github.ref, 'refs/tags/')
|
|
42
|
+
needs:
|
|
43
|
+
- build
|
|
44
|
+
runs-on: ubuntu-latest
|
|
45
|
+
environment:
|
|
46
|
+
name: pypi
|
|
47
|
+
url: https://pypi.org/p/tdmspy
|
|
48
|
+
permissions:
|
|
49
|
+
id-token: write
|
|
50
|
+
|
|
51
|
+
steps:
|
|
52
|
+
- name: Download all the distribution packages
|
|
53
|
+
uses: actions/download-artifact@v8
|
|
54
|
+
with:
|
|
55
|
+
name: python-package-distributions
|
|
56
|
+
path: dist/
|
|
57
|
+
|
|
58
|
+
- name: Publish distribution to PyPI
|
|
59
|
+
uses: pypa/gh-action-pypi-publish@release/v1
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
name: Run tests
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
- push
|
|
5
|
+
|
|
6
|
+
jobs:
|
|
7
|
+
os-python-matrix:
|
|
8
|
+
strategy:
|
|
9
|
+
matrix:
|
|
10
|
+
os:
|
|
11
|
+
- name: macos
|
|
12
|
+
icon: 🍏
|
|
13
|
+
- name: ubuntu
|
|
14
|
+
icon: 🐧
|
|
15
|
+
- name: windows
|
|
16
|
+
icon: 🪟
|
|
17
|
+
python-version:
|
|
18
|
+
- "3.13"
|
|
19
|
+
- "3.14"
|
|
20
|
+
|
|
21
|
+
runs-on: ${{ matrix.os.name }}-latest
|
|
22
|
+
name: ${{ matrix.os.icon }} Python ${{ matrix.python-version }}
|
|
23
|
+
|
|
24
|
+
steps:
|
|
25
|
+
- name: Checkout code
|
|
26
|
+
uses: actions/checkout@v7
|
|
27
|
+
|
|
28
|
+
- name: Set up Python ${{ matrix.python-version }}
|
|
29
|
+
uses: actions/setup-python@v7
|
|
30
|
+
with:
|
|
31
|
+
python-version: ${{ matrix.python-version }}
|
|
32
|
+
|
|
33
|
+
- name: Install uv
|
|
34
|
+
uses: astral-sh/setup-uv@v7
|
|
35
|
+
with:
|
|
36
|
+
python-version: ${{ matrix.python-version }}
|
|
37
|
+
|
|
38
|
+
- name: Install just
|
|
39
|
+
uses: taiki-e/install-action@just
|
|
40
|
+
|
|
41
|
+
- name: Install package
|
|
42
|
+
run: |
|
|
43
|
+
uv venv
|
|
44
|
+
uv sync
|
|
45
|
+
|
|
46
|
+
- name: Check and test code
|
|
47
|
+
run: just test
|
tdmspy-0.1.0/.gitignore
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
3.13
|
tdmspy-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Moritz Breurather, Institute of Production Engineering and Photonic Technologies TU Wien
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
tdmspy-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: tdmspy
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Pure Python reader for National Instruments TDMS files, MIT licensed
|
|
5
|
+
License-Expression: MIT
|
|
6
|
+
License-File: LICENSE
|
|
7
|
+
Requires-Python: >=3.13
|
|
8
|
+
Requires-Dist: numpy>=2.5.1
|
|
9
|
+
Description-Content-Type: text/markdown
|
|
10
|
+
|
|
11
|
+
# tdmspy
|
|
12
|
+
|
|
13
|
+
A pure Python reader for National Instruments **TDMS** files, MIT licensed. Its only runtime dependency is numpy.
|
|
14
|
+
|
|
15
|
+
It is written from the format specification and tested against real files and
|
|
16
|
+
synthetic test cases, one for every special case of the format.
|
|
17
|
+
|
|
18
|
+
## Install
|
|
19
|
+
|
|
20
|
+
```sh
|
|
21
|
+
uv venv --allow-existing
|
|
22
|
+
uv sync --all-extras
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## Usage
|
|
26
|
+
|
|
27
|
+
```python
|
|
28
|
+
import tdmspy
|
|
29
|
+
|
|
30
|
+
file = tdmspy.read("measurement.tdms") # indexes the file; no channel data is read yet
|
|
31
|
+
for group in file.groups: # in file order, not sorted
|
|
32
|
+
for channel in group.channels:
|
|
33
|
+
print(group.name, channel.name, channel.dtype, len(channel), channel.properties)
|
|
34
|
+
|
|
35
|
+
values = file["rawdata"]["Channel1"].read() # numpy array, native byte order, NI scaling applied
|
|
36
|
+
raw = file["rawdata"]["Channel1"].read(scaled=False)
|
|
37
|
+
part = file["rawdata"]["Channel1"].read(start=1000, stop=2000)
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
The details (what is returned for each data type, what is refused, how truncated files are treated) are in
|
|
41
|
+
[`doc/design.md`](doc/design.md); every special case of the format the reader has to handle is listed with the files
|
|
42
|
+
that exercise it in [`doc/special-cases.md`](doc/special-cases.md).
|
|
43
|
+
|
|
44
|
+
## Tests
|
|
45
|
+
|
|
46
|
+
```sh
|
|
47
|
+
just test # lint, then everything except the slow tests
|
|
48
|
+
just test-all # including large files and long fuzz runs
|
|
49
|
+
just testcases # rewrite the stored synthetic files in tests/data/synthetic
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
The tests are the specification of the reader:
|
|
53
|
+
|
|
54
|
+
- `tests/catalogue.py` lists every special case of the format. Add a case there first.
|
|
55
|
+
- `tests/builder.py` is a byte-level TDMS writer for the tests (interleaved, big-endian, several chunks, truncated,
|
|
56
|
+
corrupt, every property type, ...). It does not use tdmspy.
|
|
57
|
+
- `tests/testcases.py` has one small file per special case, each with what it must read as (`Truth`). The files are
|
|
58
|
+
stored in `tests/data/synthetic`; the long ones are built when a test needs them.
|
|
59
|
+
- `tests/data/real` are real files, with their identifying strings replaced by
|
|
60
|
+
placeholders; `tests/test_real_files.py` checks them and `tests/test_privacy.py` keeps identifiers out of the test data.
|
|
61
|
+
- `tests/test_reader_*.py` are the tests of `tdmspy.read`: all test cases, the API, strict mode and scaling.
|
|
62
|
+
- `tests/test_fuzz.py` and `tests/test_robust.py` damage files and read large ones.
|
|
63
|
+
|
|
64
|
+
`just catalogue` regenerates `doc/special-cases.md`.
|
|
65
|
+
|
|
66
|
+
## Inspired by
|
|
67
|
+
|
|
68
|
+
[npTDMS](https://github.com/adamreeve/npTDMS), [pyTDMS](https://github.com/petebachant/pyTDMS) and
|
|
69
|
+
[ICOschema](https://github.com/ift-tuwien/ICOschema), which tdmspy was created to aid: ICOschema reads TDMS files
|
|
70
|
+
through it. No code of any of these projects is used here.
|
|
71
|
+
|
|
72
|
+
TDMS, LabVIEW and NI are trademarks of National Instruments. tdmspy is not affiliated with or endorsed by NI.
|
|
73
|
+
|
|
74
|
+
## Licence
|
|
75
|
+
|
|
76
|
+
MIT, see [LICENSE](LICENSE).
|
tdmspy-0.1.0/README.md
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# tdmspy
|
|
2
|
+
|
|
3
|
+
A pure Python reader for National Instruments **TDMS** files, MIT licensed. Its only runtime dependency is numpy.
|
|
4
|
+
|
|
5
|
+
It is written from the format specification and tested against real files and
|
|
6
|
+
synthetic test cases, one for every special case of the format.
|
|
7
|
+
|
|
8
|
+
## Install
|
|
9
|
+
|
|
10
|
+
```sh
|
|
11
|
+
uv venv --allow-existing
|
|
12
|
+
uv sync --all-extras
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## Usage
|
|
16
|
+
|
|
17
|
+
```python
|
|
18
|
+
import tdmspy
|
|
19
|
+
|
|
20
|
+
file = tdmspy.read("measurement.tdms") # indexes the file; no channel data is read yet
|
|
21
|
+
for group in file.groups: # in file order, not sorted
|
|
22
|
+
for channel in group.channels:
|
|
23
|
+
print(group.name, channel.name, channel.dtype, len(channel), channel.properties)
|
|
24
|
+
|
|
25
|
+
values = file["rawdata"]["Channel1"].read() # numpy array, native byte order, NI scaling applied
|
|
26
|
+
raw = file["rawdata"]["Channel1"].read(scaled=False)
|
|
27
|
+
part = file["rawdata"]["Channel1"].read(start=1000, stop=2000)
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
The details (what is returned for each data type, what is refused, how truncated files are treated) are in
|
|
31
|
+
[`doc/design.md`](doc/design.md); every special case of the format the reader has to handle is listed with the files
|
|
32
|
+
that exercise it in [`doc/special-cases.md`](doc/special-cases.md).
|
|
33
|
+
|
|
34
|
+
## Tests
|
|
35
|
+
|
|
36
|
+
```sh
|
|
37
|
+
just test # lint, then everything except the slow tests
|
|
38
|
+
just test-all # including large files and long fuzz runs
|
|
39
|
+
just testcases # rewrite the stored synthetic files in tests/data/synthetic
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
The tests are the specification of the reader:
|
|
43
|
+
|
|
44
|
+
- `tests/catalogue.py` lists every special case of the format. Add a case there first.
|
|
45
|
+
- `tests/builder.py` is a byte-level TDMS writer for the tests (interleaved, big-endian, several chunks, truncated,
|
|
46
|
+
corrupt, every property type, ...). It does not use tdmspy.
|
|
47
|
+
- `tests/testcases.py` has one small file per special case, each with what it must read as (`Truth`). The files are
|
|
48
|
+
stored in `tests/data/synthetic`; the long ones are built when a test needs them.
|
|
49
|
+
- `tests/data/real` are real files, with their identifying strings replaced by
|
|
50
|
+
placeholders; `tests/test_real_files.py` checks them and `tests/test_privacy.py` keeps identifiers out of the test data.
|
|
51
|
+
- `tests/test_reader_*.py` are the tests of `tdmspy.read`: all test cases, the API, strict mode and scaling.
|
|
52
|
+
- `tests/test_fuzz.py` and `tests/test_robust.py` damage files and read large ones.
|
|
53
|
+
|
|
54
|
+
`just catalogue` regenerates `doc/special-cases.md`.
|
|
55
|
+
|
|
56
|
+
## Inspired by
|
|
57
|
+
|
|
58
|
+
[npTDMS](https://github.com/adamreeve/npTDMS), [pyTDMS](https://github.com/petebachant/pyTDMS) and
|
|
59
|
+
[ICOschema](https://github.com/ift-tuwien/ICOschema), which tdmspy was created to aid: ICOschema reads TDMS files
|
|
60
|
+
through it. No code of any of these projects is used here.
|
|
61
|
+
|
|
62
|
+
TDMS, LabVIEW and NI are trademarks of National Instruments. tdmspy is not affiliated with or endorsed by NI.
|
|
63
|
+
|
|
64
|
+
## Licence
|
|
65
|
+
|
|
66
|
+
MIT, see [LICENSE](LICENSE).
|
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
# Design and contract of tdmspy
|
|
2
|
+
|
|
3
|
+
This is what `tdmspy.read` has to do. The tests (`tests/`) check every statement here; where this file and a test
|
|
4
|
+
disagree, one of them is a bug. The list of special cases of the format is in [special-cases.md](special-cases.md).
|
|
5
|
+
|
|
6
|
+
## Goals and non-goals
|
|
7
|
+
|
|
8
|
+
- Read TDMS files (version 4712 and 4713) of any size, including those written by LabVIEW and other
|
|
9
|
+
libraries; correctly in every layout the format allows that real files use.
|
|
10
|
+
- Pure Python and numpy, MIT licensed: no native code, no dependency beyond numpy.
|
|
11
|
+
- Fast enough to open a file of a million segments and to read a 1 GB channel without copying it more than once.
|
|
12
|
+
- Honest: what is not supported is an error that says so, never silently wrong data.
|
|
13
|
+
- Non-goals: writing TDMS files, DAQmx raw data, `.tdms_index` files, LabVIEW `.tdx` / TDM, pandas or HDF5 export.
|
|
14
|
+
|
|
15
|
+
## The API
|
|
16
|
+
|
|
17
|
+
```python
|
|
18
|
+
import tdmspy
|
|
19
|
+
|
|
20
|
+
file = tdmspy.read(path, strict=False) # path: str or os.PathLike
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
`read` reads the structure of the file (every segment's lead-in and metadata) and keeps no channel data in memory.
|
|
24
|
+
The file is opened again for every `Channel.read`, so a `TdmsFile` holds no open file handle and can be used from
|
|
25
|
+
several threads.
|
|
26
|
+
|
|
27
|
+
| Object | Members |
|
|
28
|
+
|---|---|
|
|
29
|
+
| `TdmsFile` | `path`, `properties` (dict), `groups` (list, file order), `truncated` (bool); `file["name"]` returns the group (`KeyError` if missing), `"name" in file`, iterating gives the groups |
|
|
30
|
+
| `Group` | `name`, `properties`, `channels` (list, file order); `group["name"]`, `"name" in group` |
|
|
31
|
+
| `Channel` | `name`, `properties`, `dtype` (a native numpy dtype, `None` for a channel without a data type), `len(channel)`, `read(start=0, stop=None, *, scaled=True)` |
|
|
32
|
+
| `Timestamp` | `seconds` since 1904-01-01 UTC, `fraction` in units of 2**-64 s; `to_datetime64(unit)`, `from_datetime64` |
|
|
33
|
+
|
|
34
|
+
Order: groups and channels are in the order of their **first appearance** in the file (a channel's path appearing
|
|
35
|
+
counts for its group as well), never sorted. Properties are in the order they were first written; a later segment
|
|
36
|
+
that rewrites a property changes its value, not its position.
|
|
37
|
+
|
|
38
|
+
Names are exactly as stored. `''` in a path is one apostrophe; `/` inside quotes is part of the name.
|
|
39
|
+
|
|
40
|
+
### Property values
|
|
41
|
+
|
|
42
|
+
Python values: `int` for all integer types, `float` for single, double and the with-unit variants (a single
|
|
43
|
+
precision value is converted exactly), `bool`, `str`, `complex`, and `Timestamp` for timestamps (both parts kept
|
|
44
|
+
exactly). The width of a property is not kept.
|
|
45
|
+
|
|
46
|
+
### Channel data
|
|
47
|
+
|
|
48
|
+
`read` returns a numpy array in **native byte order**, whatever the file has, and a copy the caller may change.
|
|
49
|
+
|
|
50
|
+
| TDMS type | numpy |
|
|
51
|
+
|---|---|
|
|
52
|
+
| int8 ... int64, uint8 ... uint64 | the same |
|
|
53
|
+
| single, double | float32, float64 (NaN, infinities and -0.0 as stored) |
|
|
54
|
+
| boolean | bool |
|
|
55
|
+
| string | object array of `str` |
|
|
56
|
+
| timestamp | `datetime64[us]`, UTC, truncated to the microsecond (never rounded up) |
|
|
57
|
+
| complex single, double | complex64, complex128 |
|
|
58
|
+
| none (the channel never has a data type) | `dtype` is `None`, length 0, `read()` returns an empty array |
|
|
59
|
+
| extended float, anything else | `UnsupportedFeatureError` when the file is opened |
|
|
60
|
+
|
|
61
|
+
`start` and `stop` count values, like a numpy slice with non-negative bounds (negative bounds are a `ValueError`;
|
|
62
|
+
`stop` beyond the end is cut). A slice reads only the bytes it needs.
|
|
63
|
+
|
|
64
|
+
### Scaling
|
|
65
|
+
|
|
66
|
+
`read(scaled=True)` is the default. If the channel (else its group, else the file) has `NI_Number_Of_Scales` > 0 and
|
|
67
|
+
`NI_Scaling_Status` is not `scaled`, the NI scales are applied: **Linear** (`x * slope + intercept`) and
|
|
68
|
+
**Polynomial** (coefficients from `..._Polynomial_Coefficients_Size`, default 4), each reading either the raw data
|
|
69
|
+
(`..._Input_Source` = 0xFFFFFFFF, also when absent) or the output of an earlier scale; the last scale is the result.
|
|
70
|
+
The arithmetic is double precision whatever the data type, and the result is float64. Without scaling, `read`
|
|
71
|
+
returns the stored values with their stored dtype. `read(scaled=False)` is always the stored values.
|
|
72
|
+
|
|
73
|
+
Any other scale type (RTD, thermocouple, table, strain, ...) raises `UnsupportedScalingError` on a scaled read; the
|
|
74
|
+
raw read works.
|
|
75
|
+
|
|
76
|
+
### Truncated files
|
|
77
|
+
|
|
78
|
+
Writers append to the file while acquiring, and a crash or a copy can cut a file anywhere.
|
|
79
|
+
|
|
80
|
+
- A file that ends inside its last segment is read as far as it is complete: the complete values of every channel of
|
|
81
|
+
the cut segment (for contiguous layout each channel keeps the complete values it has; for interleaved layout the
|
|
82
|
+
complete rows), and nothing of a segment whose metadata is cut. `file.truncated` is `True`.
|
|
83
|
+
- Bytes at the end that are fewer than a lead-in (28 bytes) count as a cut as well.
|
|
84
|
+
- A last segment with next segment offset 0xFFFFFFFFFFFFFFFF (still being written) runs to the end of the file; it
|
|
85
|
+
is truncated only if its data ends inside a chunk.
|
|
86
|
+
- `read(path, strict=True)` raises `TdmsFormatError` instead of any of these.
|
|
87
|
+
- Damage that is not at the end (a wrong signature in the middle, an index that does not fit its segment) is always
|
|
88
|
+
a `TdmsFormatError`.
|
|
89
|
+
|
|
90
|
+
|
|
91
|
+
### Errors
|
|
92
|
+
|
|
93
|
+
All errors the package raises on purpose derive from `TdmsError`:
|
|
94
|
+
|
|
95
|
+
- `TdmsFormatError` (also a `ValueError`): not TDMS, or inconsistent: wrong signature, unknown version, a file shorter
|
|
96
|
+
than a lead-in (including 0 bytes), counts and lengths that do not fit the file, an index of 0 without a previous
|
|
97
|
+
index, a data type that changes between segments, interleaved channels with different counts, an invalid object
|
|
98
|
+
path, a `.tdms_index` file.
|
|
99
|
+
- `UnsupportedFeatureError` (also a `NotImplementedError`): valid but not read: DAQmx raw data, extended floats.
|
|
100
|
+
`UnsupportedScalingError` is one of these.
|
|
101
|
+
|
|
102
|
+
Problems of the file system (`FileNotFoundError`, `PermissionError`, ...) are not wrapped.
|
|
103
|
+
|
|
104
|
+
Fuzzed and damaged files may raise only these. They must not hang or allocate memory beyond what the file's size can
|
|
105
|
+
justify: every count and length in the file is checked against the bytes that remain before anything is allocated.
|
|
106
|
+
|
|
107
|
+
## Semantics of segments
|
|
108
|
+
|
|
109
|
+
These are the rules the implementation follows. Each has files among the test cases.
|
|
110
|
+
|
|
111
|
+
1. A file is a sequence of segments. A segment is a 28 byte lead-in (`TDSm`, table of contents, version, next segment
|
|
112
|
+
offset, raw data offset), metadata, raw data. The table of contents is little-endian; the rest of the lead-in,
|
|
113
|
+
the metadata and the data use the byte order its big-endian flag says, per segment.
|
|
114
|
+
2. The metadata is read only if the metadata flag is set. If the new object list flag is also set, the list replaces
|
|
115
|
+
the active object list (the order of objects the raw data follows); without it, the objects listed only **update**
|
|
116
|
+
what the previous list had (properties, raw data index) and the list stays as it was. Objects not listed keep their
|
|
117
|
+
index and still get their share of the raw data.
|
|
118
|
+
3. A segment without metadata continues with the active list and indexes.
|
|
119
|
+
4. The raw data of a segment starts at lead-in + 28 + raw data offset and ends at next segment offset (or the end of
|
|
120
|
+
the file when it is 0xFFFFFFFFFFFFFFFF).
|
|
121
|
+
5. A raw data index is `0xFFFFFFFF` (no data in this segment), `0` (the same type and count as the object's previous
|
|
122
|
+
index), or explicit (length, type, dimension 1, number of values; strings add the total size in bytes). A channel's
|
|
123
|
+
data type cannot change.
|
|
124
|
+
6. The raw data is a number of **chunks** (size of the data / size of one chunk), each holding every channel that has data,
|
|
125
|
+
in list order: contiguous (channel after channel) or interleaved (the interleaved flag: one value of each channel in
|
|
126
|
+
turn). Channels may have different counts when contiguous; interleaved needs equal counts. A string channel is its
|
|
127
|
+
offsets (`uint32` end positions) then the text, UTF-8.
|
|
128
|
+
7. Object paths are `/`, `/'group'` or `/'group'/'channel'`. A group exists if any path mentions it.
|
|
129
|
+
8. Property updates: the last written value wins; properties are per object and accumulate over segments.
|
|
130
|
+
|
|
131
|
+
## How it is implemented
|
|
132
|
+
|
|
133
|
+
- `tdmspy/_metadata.py` parses a segment's metadata (objects, raw data indexes, properties). Every count and length is
|
|
134
|
+
checked against the bytes that remain before anything is read or allocated.
|
|
135
|
+
- `tdmspy/_scan.py` walks the segments and keeps, per channel, a list of *blocks*: `repeat` units of `per`
|
|
136
|
+
consecutive values, `stride` bytes apart, from byte `offset`. A segment of several chunks, or interleaved data, is
|
|
137
|
+
one block; the identical segments of an acquisition merge into one (a file of 250 000 segments of one channel set
|
|
138
|
+
is a handful of blocks). A segment whose metadata equals the previous segment's metadata is not parsed again.
|
|
139
|
+
- `tdmspy/file.py` reads a slice by mapping the file and copying the wanted part of the wanted blocks into a new
|
|
140
|
+
array, so a slice costs what it returns, not what the channel holds. Byte order, booleans, complex numbers and
|
|
141
|
+
timestamps are converted while copying.
|
|
142
|
+
- `tdmspy/_scaling.py` applies the NI scales.
|
|
143
|
+
- No global state; no I/O at import; no handle kept open.
|
|
144
|
+
|
|
145
|
+
## Development
|
|
146
|
+
|
|
147
|
+
- The catalogue (`tests/catalogue.py`) lists the special cases. A new case is added there, then as a file in
|
|
148
|
+
`tests/testcases.py` with the values it must read as, and the file is stored in `tests/data/synthetic` (`just testcases`).
|
|
149
|
+
- `tests/builder.py` writes any byte sequence the format can express, including broken ones.
|
|
150
|
+
- Real files go into `tests/data/real`, listed in `REAL_FILES` with the cases they exercise; `just real-expected`
|
|
151
|
+
stores what they contain.
|
|
152
|
+
- Test data must not identify what was measured, where, by whom or on which device: see `tests/test_privacy.py`.
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
# Special cases of the TDMS format
|
|
2
|
+
|
|
3
|
+
<!-- generated by `just catalogue` from tests/catalogue.py and tests/testcases.py: do not edit -->
|
|
4
|
+
|
|
5
|
+
77 special cases, exercised by 80 synthetic files (`tests/testcases.py`), 4 real files (`tests/data/real`) and a few behaviour tests.
|
|
6
|
+
|
|
7
|
+
## The file
|
|
8
|
+
|
|
9
|
+
| Case | The reader must | Exercised by |
|
|
10
|
+
|---|---|---|
|
|
11
|
+
| `file.minimal` | A file of one segment with one group and one channel is read. | `minimal` |
|
|
12
|
+
| `file.empty_is_error` | A file of 0 bytes, or shorter than a lead-in (28 bytes), is a `TdmsFormatError`. | `empty_file`, `shorter_than_lead_in` |
|
|
13
|
+
| `file.wrong_signature_is_error` | A file that does not start with `TDSm` is a `TdmsFormatError` (not a crash, not a guess). | `wrong_signature` |
|
|
14
|
+
| `file.index_file_is_error` | A `.tdms_index` file (signature `TDSh`) is refused with a clear message: it holds no data. | `index_file` |
|
|
15
|
+
| `file.not_tdms_inside` | A segment with a wrong signature after valid ones is a `TdmsFormatError`, unless it is the cut-off end of the file. | `garbage_after_segments` |
|
|
16
|
+
|
|
17
|
+
## Versions
|
|
18
|
+
|
|
19
|
+
| Case | The reader must | Exercised by |
|
|
20
|
+
|---|---|---|
|
|
21
|
+
| `version.4712` | Segments of version 4712 (TDMS 1.0) are read. | `version_4712` |
|
|
22
|
+
| `version.4713` | Segments of version 4713 (TDMS 2.0) are read. | `minimal`, real: `one_int16_channel.tdms`, real: `three_int16_channels.tdms`, real: `three_int16_channels_short.tdms`, real: `unequal_counts_untyped_channels.tdms` |
|
|
23
|
+
| `version.unknown_is_error` | Any other version number is a `TdmsFormatError`. | `version_unknown` |
|
|
24
|
+
|
|
25
|
+
## Segments
|
|
26
|
+
|
|
27
|
+
| Case | The reader must | Exercised by |
|
|
28
|
+
|---|---|---|
|
|
29
|
+
| `segment.many` | Hundreds of small segments of one channel are read as one continuous channel, in order. | `many_segments`, `many_segments_5000`, real: `one_int16_channel.tdms`, real: `three_int16_channels.tdms`, real: `three_int16_channels_short.tdms`, real: `unequal_counts_untyped_channels.tdms` |
|
|
30
|
+
| `segment.metadata_only` | A segment without raw data (no raw data flag) defines objects and properties and adds no values. | `metadata_only_segment` |
|
|
31
|
+
| `segment.raw_only` | A segment without metadata (no metadata flag) holds more data for the previous segment's objects, layout and indexes. | `raw_only_segment` |
|
|
32
|
+
| `segment.meta_without_new_list` | A segment with metadata but without the new-object-list flag updates only the objects it lists; every other object keeps its index and still receives its share of the raw data. | `meta_without_new_list`, real: `unequal_counts_untyped_channels.tdms` |
|
|
33
|
+
| `segment.new_list_replaces` | A new object list replaces the active one: an object left out gets no more data in that segment, one added gets some. | `new_list_replaces` |
|
|
34
|
+
| `segment.raw_offset_padding` | The raw data starts where the lead-in's raw data offset says, even if the metadata is shorter. | `raw_offset_padding` |
|
|
35
|
+
| `segment.zero_values` | A channel whose index says 0 values (or a segment with an empty raw data section) adds nothing and breaks nothing. | `zero_values` |
|
|
36
|
+
| `segment.next_offset_unknown` | A last segment whose next segment offset is 0xFFFFFFFFFFFFFFFF (still being written) runs to the end of the file. | `next_offset_unknown` |
|
|
37
|
+
| `segment.truncated_file` | A file cut off inside its last segment gives the complete values of that segment and sets `file.truncated`; `strict=True` raises instead. | `truncated_mid_values`, `truncated_in_metadata`, `truncated_in_lead_in`, `truncated_interleaved_row`, `truncated_second_channel`, `truncated_after_whole_chunks` |
|
|
38
|
+
| `segment.next_offset_beyond_end` | A last segment declaring more bytes than the file has is treated as truncated. | `next_offset_beyond_end` |
|
|
39
|
+
| `segment.chunks` | A segment whose raw data is several repeats of the channel set (chunks) gives all the values, in order. | `chunks`, `truncated_after_whole_chunks`, `interleaved_chunks` |
|
|
40
|
+
| `segment.partial_chunk` | Bytes after the last whole chunk of a truncated segment give the values that are complete, none that are cut. | `truncated_mid_values`, `truncated_interleaved_row`, `truncated_second_channel` |
|
|
41
|
+
|
|
42
|
+
## The raw data index
|
|
43
|
+
|
|
44
|
+
| Case | The reader must | Exercised by |
|
|
45
|
+
|---|---|---|
|
|
46
|
+
| `index.none` | An index of 0xFFFFFFFF means the object has no data in this segment (and is not an error). | `metadata_only_segment`, `index_none_then_data`, `channel_without_data` |
|
|
47
|
+
| `index.same_as_previous` | An index of 0 means the previous segment's type and count for that channel. | `many_segments`, `index_same_after_change`, `mixed_endianness`, `property_update`, `many_segments_5000`, real: `one_int16_channel.tdms`, real: `three_int16_channels.tdms`, real: `three_int16_channels_short.tdms`, real: `unequal_counts_untyped_channels.tdms` |
|
|
48
|
+
| `index.changed_count` | A new explicit index changes a channel's count in a later segment. | `meta_without_new_list`, `zero_values`, `index_same_after_change` |
|
|
49
|
+
| `index.string` | The string index (type, dimension, count, total size) is read. | `strings` |
|
|
50
|
+
| `index.type_change_is_error` | A channel whose data type changes between segments is a `TdmsFormatError`. | `index_type_change` |
|
|
51
|
+
| `index.same_without_previous_is_error` | An index of 0 for a channel that never had an explicit one is a `TdmsFormatError`. | `index_same_without_previous` |
|
|
52
|
+
|
|
53
|
+
## The layout of the raw data
|
|
54
|
+
|
|
55
|
+
| Case | The reader must | Exercised by |
|
|
56
|
+
|---|---|---|
|
|
57
|
+
| `layout.contiguous_multi_channel` | Channels of one chunk are stored one after the other, each with its own count. | `truncated_second_channel`, `contiguous_three_types`, real: `one_int16_channel.tdms`, real: `three_int16_channels.tdms`, real: `three_int16_channels_short.tdms` |
|
|
58
|
+
| `layout.unequal_counts` | Channels in one segment may have different counts (contiguous layout). | `unequal_counts`, real: `unequal_counts_untyped_channels.tdms` |
|
|
59
|
+
| `layout.interleaved` | Interleaved data (one value of each channel after the other) is read. | `truncated_interleaved_row`, `interleaved_two_channels`, `interleaved_three_channels_floats`, `big_endian_interleaved` |
|
|
60
|
+
| `layout.interleaved_chunks` | Interleaved data in several chunks is read. | `interleaved_chunks` |
|
|
61
|
+
| `layout.interleaved_mixed_types` | Interleaved channels of different fixed sizes (int16 next to float64) are read. | `interleaved_mixed_types` |
|
|
62
|
+
| `layout.interleaved_unequal_is_error` | Interleaved channels with different counts are a `TdmsFormatError`. | `interleaved_unequal_counts` |
|
|
63
|
+
| `layout.big_endian` | A segment with the big-endian flag is read completely: lead-in, metadata and data. | `big_endian`, `big_endian_interleaved` |
|
|
64
|
+
| `layout.mixed_endianness` | Segments of different byte order in one file are each read in their own. | `mixed_endianness` |
|
|
65
|
+
| `layout.channel_added_later` | A channel that first appears in a later segment starts there. | `new_list_replaces` |
|
|
66
|
+
| `layout.channel_dropped` | A channel that stops getting data keeps what it has. | `new_list_replaces` |
|
|
67
|
+
|
|
68
|
+
## Data types
|
|
69
|
+
|
|
70
|
+
| Case | The reader must | Exercised by |
|
|
71
|
+
|---|---|---|
|
|
72
|
+
| `type.integers` | int8 to int64 and uint8 to uint64 channels are read exactly, as native numpy arrays. | `integers`, real: `one_int16_channel.tdms`, real: `three_int16_channels.tdms`, real: `three_int16_channels_short.tdms` |
|
|
73
|
+
| `type.floats` | float32 and float64 channels are read exactly, including NaN, inf and -0.0. | `floats`, real: `one_int16_channel.tdms`, real: `three_int16_channels.tdms`, real: `three_int16_channels_short.tdms`, real: `unequal_counts_untyped_channels.tdms` |
|
|
74
|
+
| `type.bool` | A boolean channel is read as a numpy bool array. | `booleans` |
|
|
75
|
+
| `type.string` | A string channel (offsets then text) is read, over several segments too, with empty and non-ASCII strings. | `strings` |
|
|
76
|
+
| `type.timestamp` | A timestamp channel is read as `datetime64[us]` (UTC, truncated). | `timestamps`, `timestamp_fraction_extremes` |
|
|
77
|
+
| `type.complex` | Complex single and double channels are read as complex64 and complex128. | `complex_numbers` |
|
|
78
|
+
| `type.no_data_type` | A channel that never has data has `dtype None`, length 0 and reads as an empty array (the quality_score channels). | `channel_without_data`, real: `unequal_counts_untyped_channels.tdms` |
|
|
79
|
+
| `type.unsupported_is_error` | A data type this package cannot read (extended float) is an `UnsupportedFeatureError`, not garbage. | `extended_float_channel` |
|
|
80
|
+
|
|
81
|
+
## Properties
|
|
82
|
+
|
|
83
|
+
| Case | The reader must | Exercised by |
|
|
84
|
+
|---|---|---|
|
|
85
|
+
| `prop.all_types` | Properties of every type (integers, floats, bool, string, timestamp, complex, with-unit floats) are read as Python values. | `property_types`, `property_complex`, `property_unit_types`, real: `one_int16_channel.tdms`, real: `three_int16_channels.tdms`, real: `three_int16_channels_short.tdms` |
|
|
86
|
+
| `prop.root_group_channel` | Properties at the file, group and channel level are kept apart. | `minimal`, `property_types`, real: `one_int16_channel.tdms`, real: `three_int16_channels.tdms`, real: `unequal_counts_untyped_channels.tdms` |
|
|
87
|
+
| `prop.update` | A property written again in a later segment has the later value; others stay. | `property_update` |
|
|
88
|
+
| `prop.added_later` | A property that first appears in a later segment is there. | `property_update` |
|
|
89
|
+
| `prop.unicode` | Names and values with non-ASCII characters are read (UTF-8). | `property_unicode` |
|
|
90
|
+
| `prop.empty_and_odd_names` | An empty string value, a name with a trailing space and a very long value are kept as they are. | `property_odd_values`, real: `unequal_counts_untyped_channels.tdms` |
|
|
91
|
+
| `prop.timestamp_exact` | A timestamp property keeps seconds and the 2**-64 fraction exactly; conversion to `datetime64` truncates. | `timestamp_fraction_extremes`, `property_types` |
|
|
92
|
+
| `prop.many` | Thousands of properties on one object are read. | `many_properties` |
|
|
93
|
+
| `prop.unsupported_type_is_error` | A property of a type this package cannot read is an `UnsupportedFeatureError`. | `property_extended_float` |
|
|
94
|
+
|
|
95
|
+
## Objects and paths
|
|
96
|
+
|
|
97
|
+
| Case | The reader must | Exercised by |
|
|
98
|
+
|---|---|---|
|
|
99
|
+
| `path.group_without_channels` | A group that only has properties is a group with no channels. | `group_without_channels`, real: `one_int16_channel.tdms`, real: `three_int16_channels.tdms`, real: `three_int16_channels_short.tdms`, real: `unequal_counts_untyped_channels.tdms` |
|
|
100
|
+
| `path.implicit_group` | A channel whose group object is never written still has its group. | `implicit_group` |
|
|
101
|
+
| `path.order` | Groups and channels are in the order of their first appearance in the file, not sorted. | `first_appearance_order` |
|
|
102
|
+
| `path.quote_and_slash` | Names with `'` (written `''`), `/` and spaces are read back as written. | `quotes_and_slashes` |
|
|
103
|
+
| `path.unicode` | Group and channel names with non-ASCII characters are read. | `property_unicode` |
|
|
104
|
+
| `path.reserved_looking_names` | Groups or channels named `info`, `name`, `groups`, `properties`, `data` or `read` are ordinary. | `reserved_looking_names`, real: `unequal_counts_untyped_channels.tdms` |
|
|
105
|
+
| `path.root_only_properties` | A file whose only object is the root has properties and no groups. | `root_only` |
|
|
106
|
+
| `path.invalid_is_error` | An object path that is not `/` or `/'group'` or `/'group'/'channel'` is a `TdmsFormatError`. | `three_level_path`, `path_without_slash` |
|
|
107
|
+
|
|
108
|
+
## Scaling
|
|
109
|
+
|
|
110
|
+
| Case | The reader must | Exercised by |
|
|
111
|
+
|---|---|---|
|
|
112
|
+
| `scale.linear` | A channel with one linear NI scale reads scaled, and raw with `scaled=False`. | `scale_linear` |
|
|
113
|
+
| `scale.polynomial` | A polynomial NI scale is applied. | `scale_polynomial` |
|
|
114
|
+
| `scale.chain` | Scales that take another scale as their input source are applied in that order. | `scale_chain` |
|
|
115
|
+
| `scale.already_scaled` | `NI_Scaling_Status` = `scaled` means the data is final and is not scaled again. | `scale_already_scaled` |
|
|
116
|
+
| `scale.group_level` | Scaling declared on the group applies to its channels without their own. | `scale_group_level` |
|
|
117
|
+
| `scale.file_level` | Scaling declared on the file applies to channels and groups without their own. | `scale_file_level` |
|
|
118
|
+
| `scale.channel_overrides_group` | A channel's own scaling wins over its group's. | `scale_channel_overrides_group` |
|
|
119
|
+
| `scale.float32_input` | Scaling single precision data is done in double precision and gives float64. | `scale_float32_input` |
|
|
120
|
+
| `scale.unsupported_is_error` | A scale type that is not implemented (RTD, thermocouple, ...) raises `UnsupportedScalingError` on a scaled read; the raw read works. | `scale_unsupported_type` |
|
|
121
|
+
| `scale.none` | A channel without scale properties reads the same scaled and raw. | `scale_channel_overrides_group`, `scale_none` |
|
|
122
|
+
|
|
123
|
+
## Unsupported layouts
|
|
124
|
+
|
|
125
|
+
| Case | The reader must | Exercised by |
|
|
126
|
+
|---|---|---|
|
|
127
|
+
| `daqmx.is_error` | DAQmx raw data (indexes 0x1269 and 0x126A) is an `UnsupportedFeatureError` that names DAQmx. | `daqmx_raw_data` |
|
|
128
|
+
|
|
129
|
+
## Robustness
|
|
130
|
+
|
|
131
|
+
| Case | The reader must | Exercised by |
|
|
132
|
+
|---|---|---|
|
|
133
|
+
| `robust.huge_counts` | Declared counts and lengths far beyond the file's size are a `TdmsFormatError` without allocating them. | `huge_object_count`, `huge_property_count`, `huge_string_length`, `huge_path_length`, `huge_value_count` |
|
|
134
|
+
| `robust.fuzz` | Truncated and randomly damaged files raise only `TdmsError` (never another exception, never hang, never allocate wildly). | `tests/test_fuzz.py::test_damaged_files_only_raise_tdms_errors` |
|
|
135
|
+
| `robust.lazy` | Opening a file reads its structure; channel data is read when asked for, and a slice reads only what it needs. | `tests/test_robust.py::test_opening_reads_the_structure_not_the_data` |
|
|
136
|
+
| `robust.large` | A file of 100 MB opens quickly and a channel reads in time proportional to its size (slow test). | `tests/test_robust.py::test_a_hundred_megabyte_channel_reads_in_proportion` |
|
|
137
|
+
| `robust.many_segments` | A file of tens of thousands of segments opens in a reasonable time. | `many_segments_5000` |
|
|
138
|
+
|
|
139
|
+
## Real files
|
|
140
|
+
|
|
141
|
+
| File | Special cases it exercises |
|
|
142
|
+
|---|---|
|
|
143
|
+
| `one_int16_channel.tdms` | `version.4713`, `segment.many`, `index.same_as_previous`, `type.integers`, `type.floats`, `prop.all_types`, `prop.root_group_channel`, `path.group_without_channels`, `layout.contiguous_multi_channel` |
|
|
144
|
+
| `three_int16_channels.tdms` | `version.4713`, `segment.many`, `index.same_as_previous`, `type.integers`, `type.floats`, `prop.all_types`, `prop.root_group_channel`, `path.group_without_channels`, `layout.contiguous_multi_channel` |
|
|
145
|
+
| `three_int16_channels_short.tdms` | `version.4713`, `segment.many`, `index.same_as_previous`, `type.integers`, `type.floats`, `prop.all_types`, `path.group_without_channels`, `layout.contiguous_multi_channel` |
|
|
146
|
+
| `unequal_counts_untyped_channels.tdms` | `version.4713`, `segment.many`, `segment.meta_without_new_list`, `index.same_as_previous`, `type.floats`, `type.no_data_type`, `path.group_without_channels`, `path.reserved_looking_names`, `layout.unequal_counts`, `prop.root_group_channel`, `prop.empty_and_odd_names` |
|