turbodata 0.1.0__tar.gz

turbodata-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or Derivative
+          Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work, excluding
+          those notices that do not pertain to any part of the Derivative
+          Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one of
+          the following places: within a NOTICE text file distributed as
+          part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and do
+          not modify the License. You may add Your own attribution notices
+          within Derivative Works that You distribute, alongside or as an
+          addendum to the NOTICE text from the Work, provided that such
+          additional attribution notices cannot be construed as modifying
+          the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 Wanjia He
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

turbodata-0.1.0/NOTICE

@@ -0,0 +1,27 @@
+turbodata
+Copyright 2026 Wanjia He
+
+This product includes software developed by Wanjia He and the turbodata
+contributors.
+
+Licensed under the Apache License, Version 2.0 (see the LICENSE file).
+
+---
+
+This software depends on third-party packages that are downloaded at build
+time and are not redistributed as part of this source tree. Each remains
+under its own license. Notable dependencies include:
+
+Go SDK (go/):
+  - github.com/foxglove/mcap/go/mcap        (MIT)
+  - github.com/klauspost/compress           (BSD-3-Clause / Apache-2.0 / MIT)
+  - github.com/vmihailenco/msgpack/v5       (BSD-2-Clause)
+  - github.com/google/go-cmp                (BSD-3-Clause)
+
+Python SDK (py/):
+  - msgpack                                 (Apache-2.0)
+  - zstandard                               (BSD-3-Clause)
+
+TypeScript SDK (ts/):
+  - @msgpack/msgpack                        (ISC)
+  - fzstd                                   (MIT)

turbodata-0.1.0/PKG-INFO

@@ -0,0 +1,361 @@
+Metadata-Version: 2.4
+Name: turbodata
+Version: 0.1.0
+Summary: Python SDK for the turbodata (.td) container format. Read/write port of the Go reference implementation.
+Author: Wanjia He
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/opheadacheh/turbodata
+Project-URL: Repository, https://github.com/opheadacheh/turbodata
+Project-URL: Issues, https://github.com/opheadacheh/turbodata/issues
+Keywords: turbodata,container-format,timeseries,robotics,mcap,msgpack,zstd
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: msgpack>=1.0.5
+Requires-Dist: zstandard>=0.22.0
+Provides-Extra: test
+Requires-Dist: pytest>=7; extra == "test"
+Dynamic: license-file
+
+# turbodata (Python)
+
+Python SDK for the turbodata `.td` container format. Read/write port of the
+Go reference implementation in `../go`. The on-disk format is identical, so
+files written by any SDK can be read by any other.
+
+Built so model developers can directly manipulate `.td` files (read training
+data, write preprocessed shards, sample at specific timestamps for multi-topic
+alignment).
+
+```bash
+pip install turbodata          # or, for local dev: pip install -e ".[test]"
+```
+
+Dependencies: `msgpack`, `zstandard`. See the
+[top-level README](../README.md) for scope and the format overview. This guide
+is API usage by feature: **write → read → sample → multi-file**. Runnable
+versions of every snippet live in [`examples/`](./examples).
+
+## Write
+
+Open one or more topics as a group, append messages with non-decreasing
+timestamps, then close the topic group. The `Writer` context manager finalizes
+the file on exit.
+
+```python
+from turbodata import Writer
+
+with open("out.td", "wb") as f, Writer(f) as w:
+    w.open_topics(
+        ["/imu", "/cam"],
+        [{"hz": 100}, {"hz": 10}],
+        compression=True,
+    )
+    w.write_message("/imu", b"...", timestamp=1_000)
+    w.write_message("/cam", b"...", timestamp=2_000)
+    w.close_topic()
+```
+
+The writer enforces non-decreasing timestamps within an open topic group.
+
+### Schema metadata
+
+Supply `schema_name`, `schema_encoding`, and binary `schema_data` per topic
+for downstream schema discovery. Missing, null, or empty values produce one
+warning per topic when opening it; writing continues by default. The minimal
+examples above omit these fields and therefore warn. Reading does not warn.
+
+```python
+from turbodata import META_KEY_SCHEMA_NAME, META_KEY_SCHEMA_ENCODING, META_KEY_SCHEMA_DATA
+
+metadata = {
+    META_KEY_SCHEMA_NAME: "Example",
+    META_KEY_SCHEMA_ENCODING: "jsonschema",
+    META_KEY_SCHEMA_DATA: b'{"type":"object"}',
+}
+# Pass metadata as the corresponding entry in open_topics' metadatas list.
+```
+
+### Chunk config
+
+Chunks are the unit of indexing and I/O. By default the writer flushes a chunk
+at a size threshold; `chunk_config` overrides the policy (by size, message
+count, or time duration).
+
+```python
+from turbodata import ChunkConfig, ChunkThresholdMode
+
+w.open_topics(
+    ["/imu"],
+    [{"hz": 100}],
+    chunk_config=ChunkConfig(mode=ChunkThresholdMode.SIZE, size=1024 * 1024),
+)
+```
+
+### Video
+
+Mark a group as video with `video=True` and append frames with
+`write_video_message` (which carries an `is_key_frame` flag). Chunk boundaries
+are gated on key frames, so a GOP is never split across two chunks.
+
+```python
+w.open_topics(["/cam/h264"], [{"codec": "h264"}], video=True)
+w.write_video_message("/cam/h264", key_frame_bytes, 1_000, True)  # first frame must be a key frame
+w.write_video_message("/cam/h264", delta_bytes, 2_000, False)
+w.close_topic()
+```
+
+A video group MUST contain exactly one topic and MUST NOT also be compressed
+(the codec already compresses the bytes). The first frame MUST be a key frame.
+
+## Read
+
+```python
+from turbodata import FileReadSource, Reader
+
+with FileReadSource("data.td") as src:
+    reader = Reader(src)
+    for msg in reader.read_messages():
+        # msg.timestamp: int (unit set by the writer)
+        # msg.topic_name: str
+        # msg.data: bytes (aliases an internal buffer; copy if you need to keep it)
+        print(msg.timestamp, msg.topic_name, len(msg.data))
+```
+
+`msg.data` aliases an internal reusable buffer and is only valid until the
+next iteration step. Pass `copy=True` for fresh bytes each iteration:
+
+```python
+for msg in reader.read_messages(copy=True):
+    ...  # msg.data is independent of subsequent pulls
+```
+
+### Order, time, and topic filters
+
+Topics not listed are skipped; chunks fully outside the `[start, end]` window
+are never fetched.
+
+```python
+from turbodata import Order
+
+reader.read_messages(
+    order=Order.REVERSE_TIME,           # default: Order.TIME
+    topic_names=["/imu", "/cam"],        # default: all topics
+    start_timestamp=1_000_000_000,       # inclusive lower bound
+    end_timestamp=2_000_000_000,         # inclusive upper bound
+)
+```
+
+### Cost-aware concurrent reads
+
+For cloud object storage (S3 / GCS) or large local reads, pass a
+`ReadStrategy`. The reader pre-plans every needed byte range, coalesces nearby
+ranges, optionally splits large reads, and issues them concurrently via a
+thread pool. Without a strategy it uses the memory-minimal lazy path.
+
+```python
+from turbodata import strategy_for_latency
+
+reader.read_messages(
+    strategy=strategy_for_latency(
+        rtt_seconds=0.020,
+        per_stream_bw=125 * 1024 * 1024,
+        concurrency=16,
+    ),
+)
+```
+
+Helper constructors mirror the Go SDK: `strategy_for_latency`,
+`strategy_for_money`, `strategy_for_blended`. Pass a hand-tuned
+`ReadStrategy(coalesce_gap=..., split_threshold=..., max_concurrency=...)`
+when you know the right values for your storage backend.
+
+### Video
+
+A non-key frame is only decodable after its GOP's key frame, so a plain time
+filter that starts mid-GOP yields bytes a decoder can't cold-start on.
+`video_decodable=True` snaps the effective start back to the latest key frame
+at or before it, so the emitted sequence starts decodable. Non-video topics
+are unaffected.
+
+```python
+reader.read_messages(
+    topic_names=["/cam/h264"],
+    start_timestamp=650,        # mid-GOP
+    video_decodable=True,       # snaps back to the key frame at/<=650
+)
+```
+
+## Sample (floor lookup at concrete timestamps)
+
+`sample` returns the floor message per `(topic, timestamp)` pair: the
+most-recent message at or before each requested timestamp. Use it for "the
+state at time T" rather than "every message in `[t0, t1]`".
+
+```python
+from turbodata import FileReadSource, Reader, SampleQuery, lin_space_timestamps
+
+with FileReadSource("data.td") as src:
+    reader = Reader(src)
+    out = reader.sample([
+        SampleQuery(topic="/imu", timestamps=lin_space_timestamps(0, 100_000_000, 10)),
+        SampleQuery(topic="/cam", timestamps=[1_000_000_000, 2_000_000_000]),
+    ])
+    for q, row in zip(["/imu", "/cam"], out):
+        for r in row:
+            if r.found:
+                print(q, r.timestamp, len(r.data))
+```
+
+`out[i][j]` corresponds to `queries[i].timestamps[j]`. The same topic must not
+appear in more than one `SampleQuery`; timestamps within one query must be
+strictly increasing.
+
+### Strategy
+
+Pass `strategy=...` to override the default sample strategy for your backend:
+
+```python
+from turbodata import ReadStrategy
+
+reader.sample(
+    queries,
+    strategy=ReadStrategy(coalesce_gap=256 * 1024, split_threshold=2 * 1024 * 1024, max_concurrency=4),
+)
+```
+
+### Video
+
+With `video_decodable=True`, results for a video topic carry a decoder-ready
+GOP sequence in `frames` (and `data` is empty) instead of the single floor
+frame. Within a row (one topic, strictly increasing timestamps), the first
+result in each GOP sets `reset_decoder=True` with the full prefix
+`[keyframe ... target]`; later results in the same GOP set
+`reset_decoder=False` with only the new frames since the previous query.
+
+```python
+out = reader.sample(
+    [SampleQuery(topic="/cam/h264", timestamps=[350, 650])],
+    video_decodable=True,
+)
+for r in out[0]:
+    if r.found:
+        # r.is_video is True; feed r.frames to a decoder,
+        # resetting it first when r.reset_decoder is True.
+        print(r.timestamp, r.reset_decoder, len(r.frames))
+```
+
+## Multi-file reading (MultiReader)
+
+`MultiReader` presents several single-file `Reader`s as one time-ordered
+stream. Read/sample options pass through to each underlying reader. Topic-name
+collisions across files are resolved with per-`Reader` `topic_remap`: names
+meant to union share an exposed name; names meant to stay distinct are remapped
+apart.
+
+```python
+from turbodata import BytesReadSource, MultiReader, Reader, SampleQuery
+
+mr = MultiReader(
+    Reader(BytesReadSource(file_a)),
+    # Keep file B's "/cam" distinct instead of unioning it with A's.
+    Reader(BytesReadSource(file_b), topic_remap={"/cam": "/cam_b"}),
+)
+
+for msg in mr.read_messages():
+    print(msg.timestamp, msg.topic_name)
+
+# sample() returns, per cell, the latest floor across the union of files.
+out = mr.sample([SampleQuery(topic="/cam", timestamps=[150, 350])])
+```
+
+### Topic remap
+
+`topic_remap` is keyed by in-file name and valued by the exposed name reported
+by `summary()`, emitted from `read_messages`, and accepted by `topic_names`
+and `SampleQuery.topic`. Names absent from the map pass through unchanged. It
+works on a single `Reader` too:
+
+```python
+reader = Reader(src, topic_remap={"/cam": "/cam_v2"})
+for msg in reader.read_messages(topic_names=["/cam_v2"]):
+    print(msg.topic_name)  # "/cam_v2"
+```
+
+The remap is validated lazily against the file's summary on first use: it
+raises `TopicRemapCollisionError` if two topics collapse onto the same exposed
+name.
+
+## ReadSource protocol
+
+A `ReadSource` exposes two methods:
+
+```python
+class ReadSource(Protocol):
+    def size(self) -> int: ...
+    def read_at(self, offset: int, n: int) -> bytes: ...
+```
+
+`read_at` MUST be safe to call from multiple threads concurrently when used
+with the cost-aware path. Built-in sources:
+
+- `FileReadSource(path_or_fd)` — local files, uses `os.pread` on POSIX
+- `BytesReadSource(bytes)` — in-memory bytes for tests/small files
+
+Roll your own for HTTP range reads, S3, GCS, etc.
+
+## Mapping to the Go SDK
+
+| Go                                  | Python                              |
+| ----------------------------------- | ----------------------------------- |
+| `NewReader(rs)`                     | `Reader(source)`                    |
+| `Summary()`                         | `reader.summary()`                  |
+| `ReadMessages(opts...)`             | `reader.read_messages(**kwargs)`    |
+| `Sample(queries, opts...)`          | `reader.sample(queries, **kwargs)`  |
+| `WithTopicNames`                    | `topic_names=[...]`                 |
+| `WithStartTimestamp`                | `start_timestamp=...`               |
+| `WithEndTimestamp`                  | `end_timestamp=...`                 |
+| `WithOrder(ReverseTimeOrder)`       | `order=Order.REVERSE_TIME`          |
+| `WithReadStrategy(s)`               | `strategy=...`                      |
+| `WithTailPrefetch(n)`               | `tail_prefetch=n`                   |
+| `WithVideoDecodable()`              | `video_decodable=True`              |
+| `WithSampleVideoDecodable()`        | `video_decodable=True` (on sample)  |
+| `WithTopicRemap(m)`                 | `Reader(src, topic_remap={...})`    |
+| `NewWriter(f)`                      | `Writer(f)`                         |
+| `OpenTopics(names, metas, opts...)` | `writer.open_topics(...)`           |
+| `WithChunkConfig(cc)`               | `chunk_config=ChunkConfig(...)`     |
+| `WithCompression()`                 | `compression=True`                  |
+| `WithVideoTopic()`                  | `video=True`                        |
+| `WriteMessage(topic, data, ts)`     | `writer.write_message(topic, data, ts)` |
+| `WriteVideoMessage(topic, data, ts, kf)` | `writer.write_video_message(topic, data, ts, kf)` |
+| `CloseTopic()`                      | `writer.close_topic()`              |
+| `Close()`                           | `writer.close()` (or `with`)        |
+
+## Run the examples
+
+Run `write_demo.py` first; it produces `examples/demo.td`, which the read and
+sample demos consume.
+
+```bash
+python examples/write_demo.py     # produces examples/demo.td (incl. a video group)
+python examples/read_demo.py      # filters, order, strategy, video snap-back
+python examples/sample_demo.py    # floor lookup, strategy, video GOP-prefix
+python examples/multiread_demo.py # union, split-via-remap, cross-file sample
+```
+
+## Run the tests
+
+```bash
+pip install -e ".[test]"
+pytest tests/
+```
+
+Cross-language tests run against the Go SDK's example fixtures
+(`../ts/test/fixtures/*.td` and `../go/examples/demo.td`) when those files
+are present.