ytclone 0.1.5__tar.gz

ytclone-0.1.5/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Alexander Vasin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ytclone-0.1.5/PKG-INFO

@@ -0,0 +1,129 @@
+Metadata-Version: 2.4
+Name: ytclone
+Version: 0.1.5
+Summary: Resumable copying of YTsaurus tables, files and directories to and from S3
+Author: Alexander Vasin
+Author-email: Alexander Vasin <hi@alvass.in>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS :: MacOS X
+Requires-Dist: aiochannel>=1.3,<2
+Requires-Dist: aiohttp>=3.14.3,<4
+Requires-Dist: aiohttp-s3-client>=1.0.14,<2
+Requires-Dist: aiomisc>=17.10.3,<19
+Requires-Dist: argclass>=1.11.1,<2
+Requires-Dist: ytsaurus-client>=0.13.54,<0.14
+Requires-Dist: ytsaurus-yson>=0.4.10,<0.5
+Requires-Dist: zstandard>=0.25.0,<1
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+# ytclone
+
+Copy static tables, files, and directories between YTsaurus and S3, preserving
+schemas and portable attributes. Repeat a command to skip completed, unchanged
+nodes and retry unfinished copies.
+
+Restoring from S3 requires an archive created by `ytclone`.
+
+## Installation
+
+Install from PyPI in your Python virtual environment:
+
+```bash
+pip install ytclone
+```
+
+Requires **CPython 3.10+** on **Linux x86_64 with glibc** or **macOS ARM64**
+(Apple Silicon). Other platforms are currently unsupported because the required
+[`ytsaurus-yson`](https://pypi.org/project/ytsaurus-yson/) dependency has no wheels
+for them.
+
+## Quick start
+
+Configure access to YTsaurus and S3:
+
+```bash
+export YT_PROXY="https://yt.example.com"
+export YT_TOKEN="..."
+export AWS_ACCESS_KEY_ID="..."
+export AWS_SECRET_ACCESS_KEY="..."
+export AWS_DEFAULT_REGION="us-east-1"
+```
+
+The S3 bucket must already exist. For an S3-compatible service, also set
+`S3_ENDPOINT` to its URL. Use `AWS_SESSION_TOKEN` for temporary credentials.
+
+**Copy to S3:**
+
+```bash
+ytclone //home/analytics/events s3://my-bucket/export --log-file export.log
+```
+
+The full YT path is appended to the S3 prefix: this command stores `events` at
+`s3://my-bucket/export/home/analytics/events/`.
+
+**Restore to YTsaurus:**
+
+```bash
+ytclone s3://my-bucket/export/home/analytics/events \
+  //home/analytics/events_copy --log-file restore.log
+```
+
+The restore destination is used exactly as given: the children of `events` go
+directly under `//home/analytics/events_copy`. The same commands work for a single
+table or file. With `--log-file`, progress appears on the console and detailed
+logs are appended to the file.
+
+## Repeated runs
+
+Run the same command again after an interruption or a source update. Completed,
+unchanged nodes are skipped. New, changed, or incomplete nodes are copied again;
+an interrupted table or file starts from the beginning, not from its last part.
+
+Source data is never deleted. Destination nodes absent from the source are kept.
+Use a separate S3 prefix for each source cluster.
+
+## Parallel export
+
+For trees with many tables or files, use multiple copy processes:
+
+```bash
+ytclone-parallel //home/analytics/events s3://my-bucket/export \
+  --processes 2 --concurrency 2 --log-file export.log
+```
+
+This runs two processes with two transfers each, prioritizing directories and
+small nodes. One table still occupies one transfer slot. Parallel mode supports
+YT → S3; restore its archives with `ytclone` as above.
+
+## Before you copy
+
+- **Supported node types:** static tables, files, and directories. Dynamic tables
+  are skipped and reported; links are unsupported.
+- **One current S3 copy.** There is no archive version history. Replacing a node
+  makes its archive unavailable until completion; a failed replacement can lose
+  the previous complete copy.
+- **Permissions are not copied.** ACLs, ownership, and account settings are
+  excluded. Replacing an existing YT table or file recreates the node, so its
+  previous settings are not preserved either.
+- **Consistency is per node.** A directory tree is not a single point-in-time
+  snapshot.
+- **Avoid overlapping writers.** Do not run commands that write to the same
+  destination nodes concurrently, or restore an archive while it is being replaced.
+
+## Documentation
+
+Run `ytclone --help` or `ytclone-parallel --help` for all command-line options.
+
+- [User guide](https://github.com/alvassin/ytclone/blob/main/USAGE.md): paths,
+  configuration, progress, retries, and parallel export.
+- [Architecture](https://github.com/alvassin/ytclone/blob/main/ARCHITECTURE.md):
+  archive format, change detection, and execution diagrams.
+- [Contributing](https://github.com/alvassin/ytclone/blob/main/CONTRIBUTING.md):
+  installation from source, development, and tests.
+
+## License
+
+[MIT](https://github.com/alvassin/ytclone/blob/main/LICENSE).

ytclone-0.1.5/README.md

@@ -0,0 +1,108 @@
+# ytclone
+
+Copy static tables, files, and directories between YTsaurus and S3, preserving
+schemas and portable attributes. Repeat a command to skip completed, unchanged
+nodes and retry unfinished copies.
+
+Restoring from S3 requires an archive created by `ytclone`.
+
+## Installation
+
+Install from PyPI in your Python virtual environment:
+
+```bash
+pip install ytclone
+```
+
+Requires **CPython 3.10+** on **Linux x86_64 with glibc** or **macOS ARM64**
+(Apple Silicon). Other platforms are currently unsupported because the required
+[`ytsaurus-yson`](https://pypi.org/project/ytsaurus-yson/) dependency has no wheels
+for them.
+
+## Quick start
+
+Configure access to YTsaurus and S3:
+
+```bash
+export YT_PROXY="https://yt.example.com"
+export YT_TOKEN="..."
+export AWS_ACCESS_KEY_ID="..."
+export AWS_SECRET_ACCESS_KEY="..."
+export AWS_DEFAULT_REGION="us-east-1"
+```
+
+The S3 bucket must already exist. For an S3-compatible service, also set
+`S3_ENDPOINT` to its URL. Use `AWS_SESSION_TOKEN` for temporary credentials.
+
+**Copy to S3:**
+
+```bash
+ytclone //home/analytics/events s3://my-bucket/export --log-file export.log
+```
+
+The full YT path is appended to the S3 prefix: this command stores `events` at
+`s3://my-bucket/export/home/analytics/events/`.
+
+**Restore to YTsaurus:**
+
+```bash
+ytclone s3://my-bucket/export/home/analytics/events \
+  //home/analytics/events_copy --log-file restore.log
+```
+
+The restore destination is used exactly as given: the children of `events` go
+directly under `//home/analytics/events_copy`. The same commands work for a single
+table or file. With `--log-file`, progress appears on the console and detailed
+logs are appended to the file.
+
+## Repeated runs
+
+Run the same command again after an interruption or a source update. Completed,
+unchanged nodes are skipped. New, changed, or incomplete nodes are copied again;
+an interrupted table or file starts from the beginning, not from its last part.
+
+Source data is never deleted. Destination nodes absent from the source are kept.
+Use a separate S3 prefix for each source cluster.
+
+## Parallel export
+
+For trees with many tables or files, use multiple copy processes:
+
+```bash
+ytclone-parallel //home/analytics/events s3://my-bucket/export \
+  --processes 2 --concurrency 2 --log-file export.log
+```
+
+This runs two processes with two transfers each, prioritizing directories and
+small nodes. One table still occupies one transfer slot. Parallel mode supports
+YT → S3; restore its archives with `ytclone` as above.
+
+## Before you copy
+
+- **Supported node types:** static tables, files, and directories. Dynamic tables
+  are skipped and reported; links are unsupported.
+- **One current S3 copy.** There is no archive version history. Replacing a node
+  makes its archive unavailable until completion; a failed replacement can lose
+  the previous complete copy.
+- **Permissions are not copied.** ACLs, ownership, and account settings are
+  excluded. Replacing an existing YT table or file recreates the node, so its
+  previous settings are not preserved either.
+- **Consistency is per node.** A directory tree is not a single point-in-time
+  snapshot.
+- **Avoid overlapping writers.** Do not run commands that write to the same
+  destination nodes concurrently, or restore an archive while it is being replaced.
+
+## Documentation
+
+Run `ytclone --help` or `ytclone-parallel --help` for all command-line options.
+
+- [User guide](https://github.com/alvassin/ytclone/blob/main/USAGE.md): paths,
+  configuration, progress, retries, and parallel export.
+- [Architecture](https://github.com/alvassin/ytclone/blob/main/ARCHITECTURE.md):
+  archive format, change detection, and execution diagrams.
+- [Contributing](https://github.com/alvassin/ytclone/blob/main/CONTRIBUTING.md):
+  installation from source, development, and tests.
+
+## License
+
+[MIT](https://github.com/alvassin/ytclone/blob/main/LICENSE).

ytclone-0.1.5/pyproject.toml

@@ -0,0 +1,40 @@
+[project]
+name = "ytclone"
+version = "0.1.5"
+description = "Resumable copying of YTsaurus tables, files and directories to and from S3"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.10"
+classifiers = [
+    "Operating System :: POSIX :: Linux",
+    "Operating System :: MacOS :: MacOS X",
+]
+dependencies = [
+    "aiochannel>=1.3,<2",
+    "aiohttp>=3.14.3,<4",
+    "aiohttp-s3-client>=1.0.14,<2",
+    "aiomisc>=17.10.3,<19",
+    "argclass>=1.11.1,<2",
+    "ytsaurus-client>=0.13.54,<0.14",
+    "ytsaurus-yson>=0.4.10,<0.5",
+    "zstandard>=0.25.0,<1",
+]
+
+[[project.authors]]
+name = "Alexander Vasin"
+email = "hi@alvass.in"
+
+[project.scripts]
+ytclone = "ytclone:main"
+ytclone-parallel = "ytclone.parallel.cli:main"
+
+[build-system]
+requires = ["uv_build>=0.12.19,<0.13.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=8",
+    "ruff>=0.16",
+]

ytclone-0.1.5/pyproject.toml.orig

@@ -0,0 +1,38 @@
+[project]
+name = "ytclone"
+version = "0.1.5"
+description = "Resumable copying of YTsaurus tables, files and directories to and from S3"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [
+    { name = "Alexander Vasin", email = "hi@alvass.in" }
+]
+requires-python = ">=3.10"
+# Native ytsaurus-yson wheels limit support to Linux x86_64 (glibc)
+# and macOS ARM64. See README.md for installation requirements.
+classifiers = [
+    "Operating System :: POSIX :: Linux",
+    "Operating System :: MacOS :: MacOS X",
+]
+dependencies = [
+    "aiochannel>=1.3,<2",
+    "aiohttp>=3.14.3,<4",
+    "aiohttp-s3-client>=1.0.14,<2",
+    "aiomisc>=17.10.3,<19",
+    "argclass>=1.11.1,<2",
+    "ytsaurus-client>=0.13.54,<0.14",
+    "ytsaurus-yson>=0.4.10,<0.5",
+    "zstandard>=0.25.0,<1",
+]
+
+[project.scripts]
+ytclone = "ytclone:main"
+ytclone-parallel = "ytclone.parallel.cli:main"
+
+[build-system]
+requires = ["uv_build>=0.12.19,<0.13.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = ["pytest>=8", "ruff>=0.16"]

ytclone-0.1.5/src/ytclone/__init__.py

@@ -0,0 +1,8 @@
+"""Resumable YTsaurus / S3 copying."""
+
+
+def main() -> int:
+    """Load and run the command-line entry point."""
+    from .cli import main as cli_main
+
+    return cli_main()

ytclone-0.1.5/src/ytclone/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())

ytclone-0.1.5/src/ytclone/cli.py

@@ -0,0 +1,277 @@
+"""Copy YTsaurus static tables, files and directories to or from an S3 archive."""
+
+import asyncio
+import logging
+import math
+import sys
+import time
+from contextlib import nullcontext, suppress
+
+import aiohttp
+import aiomisc
+import argclass
+from aiohttp_s3_client import S3Client
+from aiomisc.log import LogFormat
+
+from .engine import CopyOptions, export_tree, restore_tree
+from .format import MIB, parse_s3_url, path_components
+from .logging import file_logging
+from .memory import estimate_memory_mib
+from .progress import Progress, ProgressReporter
+from .s3 import S3Storage
+from .yt import YtAdapter
+
+log = logging.getLogger(__name__)
+LOG_DATE_FORMAT = "%Y-%m-%d %H:%M:%S %z"
+
+
+class Arguments(argclass.Parser):
+    source: str = argclass.Argument(
+        "source", help="Source //YT/path or s3://bucket/prefix"
+    )
+    destination: str = argclass.Argument(
+        "destination", help="Destination in the other storage"
+    )
+    yt_proxy: str = argclass.Argument(env_var="YT_PROXY", required=True)
+    no_proxy_discovery: bool = argclass.Argument(
+        default=False, help="Use only the supplied YT proxy (for local Docker)"
+    )
+    endpoint: str = argclass.Argument(
+        env_var="S3_ENDPOINT", default="https://s3.amazonaws.com"
+    )
+    region: str = argclass.Argument(env_var="AWS_DEFAULT_REGION", default="us-east-1")
+    concurrency: int = argclass.Argument(
+        default=4, help="Maximum concurrent table/file transfers"
+    )
+    queue_size: int = argclass.Argument(default=16, help="Maximum queued node paths")
+    part_size_mib: int = argclass.Argument(
+        default=256,
+        help="Target uncompressed size per archive object; approximate for tables",
+    )
+    buffer_size_mib: int = argclass.Argument(
+        default=32, help="S3 multipart buffer per transfer, 5..512 MiB"
+    )
+    compression_level: int = argclass.Argument(
+        default=3, help="Zstandard level, -7..22"
+    )
+    log_file: str | None = argclass.Argument(
+        default=None,
+        type=str,
+        help="Append detailed logs to this file and show progress on stdout",
+    )
+    progress_interval: float = argclass.Argument(
+        default=5.0,
+        help="Seconds between console summaries with --log-file",
+    )
+    log_level: str = argclass.Argument(
+        default="info", choices=("debug", "info", "warning", "error")
+    )
+    log_format: str = argclass.Argument(
+        default=LogFormat.default(),
+        choices=LogFormat.choices(),
+        env_var="AIOMISC_LOG_FORMAT",
+    )
+
+
+def direction(*, source: str, destination: str) -> tuple[bool, str, str, str]:
+    """Validate paths and resolve direction, YT path, bucket and prefix."""
+    export = source.startswith("//") and destination.startswith("s3://")
+    restore = source.startswith("s3://") and destination.startswith("//")
+    if not (export or restore):
+        raise ValueError("Specify one YT path and one S3 URL; source comes first")
+    yt_path = source if export else destination
+    path_components(yt_path)
+    if any(c in yt_path for c in "\\[]{}<>&*@"):
+        raise ValueError(
+            "YT selectors, links and escaped/special node names are not supported"
+        )
+    bucket, prefix = parse_s3_url(destination if export else source)
+    return export, yt_path, bucket, prefix
+
+
+async def run(*, arguments: Arguments, options: CopyOptions) -> int:
+    """Create adapters, run the copy and return its exit code."""
+    export, yt_path, bucket, prefix = direction(
+        source=arguments.source, destination=arguments.destination
+    )
+
+    memory_budget = estimate_memory_mib(
+        concurrency=options.concurrency,
+        chunk_size=options.chunk_size,
+        compression_level=options.compression_level,
+        export=export,
+    )
+    log.info(
+        "copy_started direction=%s concurrency=%d buffer_size_mib=%d "
+        "memory_estimate_mib=%d",
+        "yt-to-s3" if export else "s3-to-yt",
+        options.concurrency,
+        options.chunk_size // MIB,
+        memory_budget,
+    )
+    log.info(
+        "Memory estimate is a planning budget; large rows, metadata and "
+        "allocator overhead can exceed it."
+    )
+
+    progress = Progress() if arguments.log_file else None
+    if progress is not None:
+        print(
+            f"Detailed log: {arguments.log_file}; "
+            f"RAM estimate: {memory_budget} MiB (planning budget, not a limit)",
+            flush=True,
+        )
+    reporter = (
+        ProgressReporter(
+            progress=progress,
+            concurrency=options.concurrency,
+            interval=arguments.progress_interval,
+            export=export,
+            stream=sys.stdout,
+        )
+        if progress is not None
+        else nullcontext()
+    )
+    started = time.monotonic()
+
+    async with reporter:
+        yt = YtAdapter(
+            proxy=arguments.yt_proxy,
+            proxy_discovery=not arguments.no_proxy_discovery,
+            progress=progress,
+        )
+        estimate_task = (
+            asyncio.create_task(estimate_total(yt=yt, path=yt_path, progress=progress))
+            if export and progress is not None
+            else None
+        )
+        try:
+            async with aiohttp.ClientSession(
+                timeout=aiohttp.ClientTimeout(
+                    total=None, sock_connect=30, sock_read=300
+                )
+            ) as session:
+                client = S3Client(
+                    session=session,
+                    url=arguments.endpoint.rstrip("/") + "/" + bucket,
+                    region=arguments.region,
+                )
+                storage = S3Storage(
+                    client=client,
+                    chunk_size=options.chunk_size,
+                    progress=progress,
+                )
+                if export:
+                    result = await export_tree(
+                        yt=yt,
+                        storage=storage,
+                        source=yt_path,
+                        prefix=prefix,
+                        cluster=arguments.yt_proxy.rstrip("/"),
+                        options=options,
+                        progress=progress,
+                    )
+                else:
+                    result = await restore_tree(
+                        yt=yt,
+                        storage=storage,
+                        source_prefix=prefix,
+                        destination=yt_path,
+                        options=options,
+                        progress=progress,
+                    )
+        finally:
+            if estimate_task is not None:
+                estimate_task.cancel()
+                with suppress(asyncio.CancelledError):
+                    await estimate_task
+    log.info(
+        "copy_finished completed=%d skipped=%d failed=%d seconds=%.3f",
+        result.completed,
+        result.skipped,
+        result.failed,
+        time.monotonic() - started,
+    )
+    return 1 if result.failed else 0
+
+
+async def estimate_total(*, yt: YtAdapter, path: str, progress: Progress) -> None:
+    """Fetch optional subtree counts and storage size alongside discovery."""
+    try:
+        async with yt.worker() as worker:
+            estimate = await worker.estimate_source(path=path)
+            progress.estimated_total = estimate.node_count
+            progress.estimated_source_bytes = estimate.disk_size
+        log.info(
+            "source_estimate path=%s nodes=%s disk_space=%s",
+            path,
+            estimate.node_count,
+            estimate.disk_size,
+        )
+    except Exception:
+        log.warning("source_estimate_unavailable path=%s", path, exc_info=True)
+
+
+def main() -> int:
+    """Parse arguments and run the CLI with configured logging."""
+    arguments = Arguments(description=__doc__)
+    arguments.parse_args()
+    try:
+        if (
+            not math.isfinite(arguments.progress_interval)
+            or arguments.progress_interval <= 0
+        ):
+            raise ValueError("Progress interval must be positive and finite")
+        direction(source=arguments.source, destination=arguments.destination)
+        options = CopyOptions(
+            concurrency=arguments.concurrency,
+            queue_size=arguments.queue_size,
+            part_size=arguments.part_size_mib * MIB,
+            chunk_size=arguments.buffer_size_mib * MIB,
+            compression_level=arguments.compression_level,
+        )
+    except ValueError as exc:
+        arguments.create_parser().error(str(exc))
+
+    # aiomisc's plain formatter ignores dates; stream supports timestamps.
+    log_format = arguments.log_format
+    if log_format == LogFormat.plain.name:
+        log_format = LogFormat.stream.name
+
+    # System log collectors supply their own timestamps.
+    log_date_format = (
+        None
+        if log_format in (LogFormat.syslog.name, LogFormat.journald.name)
+        else LOG_DATE_FORMAT
+    )
+
+    log_context = (
+        file_logging(
+            filename=arguments.log_file,
+            level=arguments.log_level,
+            log_format=log_format,
+            date_format=LOG_DATE_FORMAT,
+        )
+        if arguments.log_file
+        else nullcontext()
+    )
+    try:
+        with log_context:
+            try:
+                return aiomisc.run(
+                    run(arguments=arguments, options=options),
+                    pool_size=max(4, arguments.concurrency + 1),
+                    log_level=arguments.log_level,
+                    log_format=log_format,
+                    log_date_format=log_date_format,
+                    log_config=not bool(arguments.log_file),
+                )
+            except (KeyboardInterrupt, asyncio.CancelledError):
+                log.warning("copy_cancelled")
+                return 130
+            except Exception:
+                log.exception("copy_failed")
+                return 1
+    except OSError as exc:
+        print(f"Cannot write log file {arguments.log_file}: {exc}", file=sys.stderr)
+        return 1