dagnam 0.7.0__tar.gz

dagnam-0.7.0/.gitignore

@@ -0,0 +1,236 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+coverage/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+coverage.json
+coverage*.json
+coverage-run*.log
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+#poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+#pdm.lock
+#pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+#pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Abstra
+# Abstra is an AI-powered process automation framework.
+# Ignore directories containing user credentials, local state, and settings.
+# Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#  that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#  and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#  you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+
+# Ruff stuff:
+.ruff_cache/
+
+# Import Linter
+.import_linter_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Cursor
+#  Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to
+#  exclude from AI features like autocomplete and code analysis. Recommended for sensitive data
+#  refer to https://docs.cursor.com/context/ignore-files
+.cursorignore
+.cursorindexingignore
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# Skills
+.understand-anything/
+graphify-out/
+.graphify_uncached.txt
+
+# AI Agents
+.kiro/
+.claude/
+.codex-logs/
+.agents/
+
+# IDE
+.vscode/
+
+# Local tooling logs and editor/CI temp files (never commit)
+*.log.txt
+.lint-log*
+.test*-log*
+*.yml.tmp.*
+*.yaml.tmp.*
+.tmp-*

dagnam-0.7.0/CHANGELOG.md

@@ -0,0 +1,257 @@
+# Changelog
+
+All notable changes to the `dagnam` Python SDK are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project follows [Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [0.7.0] - 2026-07-13
+
+### Added
+
+- **Automatic retries for transient failures.** Retry-safe API requests
+  (idempotent methods, plus any request carrying an idempotency key) now retry
+  on connection errors and `429`/`5xx` responses using equal-jitter exponential
+  backoff, bounded by a per-client **retry budget** (token bucket) so a flapping
+  backend cannot trigger an unbounded retry storm. A server `Retry-After` header
+  is honored but capped, so a hostile value cannot wedge the client in a long
+  sleep.
+- **Idempotency keys** for retry-safe writes: a retriable `POST` mints one
+  `uuid4` `Idempotency-Key` and reuses it across retries, so a retried create is
+  not applied twice server-side.
+- **Cross-process cache locking.** Dataset/checkpoint cache writes and LRU
+  eviction are serialized with a file lock (new `filelock>=3.13` base
+  dependency), so multiple processes sharing a cache root no longer race or
+  corrupt entries.
+- Async parity: `dagnam.aio.AsyncDagnamClient` now applies the same
+  retry/backoff and transport-error handling as the sync client, fully
+  non-blocking on the event loop.
+- `dagnam.ResponseError` — a public `APIError` subclass for malformed,
+  undecodable, or wrong-shape server response bodies.
+- Library logging contract: a package-level `NullHandler`, namespaced child
+  loggers (`dagnam.http`/`dagnam.cache`/`dagnam.lro`/`dagnam.sse`) with a
+  credential-redacting filter, and `dagnam.enable_debug_logging()` convenience.
+
+### Changed
+
+- `response_json_value`/`response_json_object`/`response_json_array` and
+  `BaseDagnamClient._expect_object` now raise `ResponseError` (an `APIError`
+  subclass) instead of a raw `TypeError`/`ValueError`/`json.JSONDecodeError`
+  when a server response body is malformed, undecodable, or the wrong JSON
+  shape. This affects every sync client method that decodes a response body
+  (datasets, training, account, hub, projects, deployments, codegen). Code
+  that narrowly caught `except TypeError`/`except ValueError` around these
+  calls should catch `except dagnam.ResponseError` (or the broader
+  `dagnam.APIError`/`dagnam.DagnamError`) instead. Client mixins that
+  optionally fall back to a plain-text response body preserve that behavior.
+
+### Security
+
+- **Path traversal / arbitrary-file-write (critical).** A server-supplied
+  dataset filename (dataset metadata / `Content-Disposition`) is now reduced to
+  a safe bare basename before it is joined under the download directory.
+  Previously a malicious or compromised server could return an absolute path or
+  a `..` sequence and make a dataset download write outside the cache — a
+  remote-code-execution vector (for example, overwriting a shell rc file).
+- **Presigned-URL credential redaction.** The log/error redaction filter now
+  scrubs presigned-URL credential parameters (`X-Amz-Signature`,
+  `X-Amz-Credential`, `Signature`, `sig`, `AWSAccessKeyId`, …) in addition to
+  `token`/`api_key`/`signature`, and transport-error text is scrubbed before it
+  reaches a log record or exception — so a presigned dataset/checkpoint URL can
+  no longer leak its signature into logs.
+- **Cache trust boundary.** The SDK warns once when the cache root is
+  group/world-writable (a same-size cache-poisoning risk on a shared host) and
+  documents `verify=True` to force a full checksum re-verification on load for
+  shared caches.
+- The `LongRunningOperation` poll interval is floored at 0.1 s, so a hostile
+  server flooding `429`/`503` cannot drive a `sleep(0)` busy-loop.
+
+## [0.6.0] - 2026-07-02
+
+### Added
+
+- Full public **exception hierarchy** re-exported from the top-level package and
+  a new `dagnam.exceptions` module, so callers can write `except dagnam.APIError`
+  (or `except dagnam.DagnamError`) without reaching into the private
+  `dagnam._core.exceptions` path.
+- `numpy` and `Pillow` are now declared **base dependencies** (they are imported
+  eagerly by the dataset layer / image loaders), so a plain `pip install dagnam`
+  can load datasets instead of failing with a bare `ModuleNotFoundError`.
+- `torchvision` is now part of the `pytorch` and `all` extras — the PyTorch
+  image-folder loaders require it, and the README already promised it.
+- Cross-platform **Agent Skill** for AI coding agents (Claude Code and Codex),
+  shipped as package data and activated with `dagnam agent install`
+  (auto-detect & prompt; `--claude` / `--codex` / `--all` / `--yes` / `--symlink`
+  for explicit and CI installs; `dagnam agent uninstall` to remove). It installs a
+  single write-once skill (`SKILL.md` + on-demand domain `reference/*.md` + helper
+  `scripts/`) into each harness's auto-discovered skills directory, plus per-harness
+  adapters: a Claude plugin with the `dagnam-runner` subagent and a `PreToolUse`
+  guard hook, and a Codex `openai.yaml` + idempotent guard-hook merge into
+  `~/.codex/hooks.json`. Skill/adapter versions are stamped to the installed SDK.
+- **Dry-run / preview-by-default guardrail** for costly, irreversible, or public
+  actions (training-job and deployment creation, deletes, hub publish): a behavioral
+  gate in the skill plus a fail-open cross-platform `PreToolUse` deny hook
+  (`python -m dagnam._agent.guardhook`).
+- Claude plugin marketplace listing (`.claude-plugin/marketplace.json`) as an
+  alternate `/plugin install` door.
+- `dagnam codegen download` now accepts a **directory** as the destination and
+  auto-extracts the generated files into it (the SDK `download(dest=<dir>)` returns
+  the directory `Path`); previously passing a directory raised a `PermissionError`.
+- Uniform CLI output convention: `codegen validate`/`preview`/`generate` and
+  `projects architecture` now accept `--json` (JSON to stdout) and `--output PATH`
+  (JSON to file), with human-readable summaries by default — matching the other
+  command groups.
+
+### Changed
+
+- **BREAKING (SDK/SSE):** `open_training_stream`, `open_deployment_stream`, and
+  async `stream_training_events` now mint short-lived, resource-scoped
+  stream-access tokens per (re)connect and send streams `?token=...`; the SDK no
+  longer sends long-lived `?api_key=` on SSE URLs. Direct consumers of the SSE
+  endpoints must switch to `POST .../stream-access-token` before opening a
+  stream, and the backend + SDK must deploy together.
+- **BREAKING (CLI):** `dagnam codegen download --output PATH` is renamed to
+  `--dest PATH` (no alias). `--output` now uniformly means "write the JSON result
+  to a file"; `--dest` is the downloaded-artifact destination (a file path streams
+  the ZIP; a directory auto-extracts). Migrate `codegen download ... --output X`
+  to `... --dest X`.
+
+- Internal: the quality gates are now clean at zero — `ruff` reports no errors
+  and `pyright` (strict, minus the unavoidable untyped-ML-library completeness
+  diagnostics) reports no errors. Remaining suppressions are centralized and
+  documented in `pyproject.toml`. Async checkpoint/codegen downloads now write
+  to disk via `asyncio.to_thread` so they no longer block the event loop, and
+  `dagnam.data.loaders` submodules are imported lazily via `__getattr__`
+  (PEP 562). No public API or runtime behavior changed.
+- Widened `requires-python` to `>=3.12` (no upper cap) and declared Python 3.13
+  support; corrected dependency floors so extras are actually installable
+  (`tensorflow>=2.16`, the first Python-3.12-capable TF; `torch>=2.4` for the
+  `pytorch` extra).
+- Label encoding is unified across `to_arrays()` and every framework loader via
+  one canonical string-identity mapping, so an integer label column no longer
+  crashes `to_pytorch_loader()` / `to_tensorflow_dataset()` / `to_flax_dataset()`
+  with a bare `KeyError`.
+
+### Fixed
+
+- The PyTorch native-numpy validation split no longer leaks the **entire** train
+  set into "val" when the validation ratio rounds down to zero.
+- Cache size/info scans skip a file removed mid-scan (concurrent eviction)
+  instead of aborting with `FileNotFoundError`.
+- A corrupt (non-integer) `max_cache_size` config value no longer crashes a
+  freshly completed download; it falls back to the default eviction budget.
+- `to_arrays()` builds an object array for ragged (variable-length) features
+  instead of raising `ValueError` under numpy ≥ 1.24.
+
+### Security
+
+- System-dataset `.npz` decoding now uses `allow_pickle=False` and refuses
+  pickled object arrays — closing an arbitrary-code-execution vector where a
+  crafted dataset could run code inside `numpy.load`.
+- Generated-code ZIP extraction is hardened against **zip-slip**: archive
+  members with path-traversal, absolute, or symlink paths are rejected instead
+  of being written outside the destination directory.
+- Checkpoint downloads that arrive without a server checksum (e.g. an S3
+  presigned redirect) are now flagged with a loud warning instead of being
+  silently accepted unverified.
+
+## [0.5.0] - 2026-06-03
+
+### Added
+
+- Release-process documentation for maintainers.
+- CLI version and account inspection commands: `dagnam --version`, `-v`,
+  `version`, `whoami`, `logout`, and read-only `config list` / `config get`.
+- Expanded CLI help text with command descriptions, examples, and argument
+  descriptions.
+- Training-job lifecycle SDK + CLI: `dagnam training create/list/get/cancel/
+  delete/logs/metrics/metrics-summary`, including after-the-fact retrieval of
+  historical logs and metrics.
+- `dagnam usage` command and `dagnam.account.*` helpers for plan, entitlement,
+  storage-quota, and per-API-key usage inspection.
+- Width-aware table rendering, pagination footers, and consistent
+  `--json` / `--output` flags across table-printing commands.
+- `--no-progress` flag for `dagnam codegen download` (progress is also hidden
+  automatically when stderr is not a TTY), matching `dagnam dataset download`.
+
+### Changed
+
+- Set the supported Python runtime to Python 3.12 so `dagnam[all]` includes
+  every optional integration.
+- CLI/SDK error messages now unwrap FastAPI `{"detail": ...}` bodies (including
+  422 validation arrays) into concise human-readable text instead of raw JSON.
+- **Breaking (SDK):** `dagnam.download_checkpoint(job_id)` with no
+  `checkpoint_id` now selects the **latest** checkpoint by epoch/step (was
+  best-then-latest). Pass `prefer_best=True` to restore the previous behavior.
+  The `dagnam checkpoint download <job> best` CLI keyword maps to
+  `prefer_best=True`.
+
+### Changed
+
+- Internal: the `load_dataset` implementation module moved from
+  `dagnam._core.load` to `dagnam.data.load` so it sits in the `data` layer it
+  composes (cache, dataset adapters, system loaders), enforcing the
+  `cli -> resources -> data -> _core` import contract. The public entry point
+  `dagnam.load_dataset` is unchanged; only the internal module path moved.
+
+### Removed
+
+- Internal compatibility-shim modules that only forwarded old import paths:
+  the `dagnam.services` package, `dagnam._core._common` / `_resolver` / `_sse`,
+  the `dagnam.data.loaders.*_loader` forwarders (`audio_loader`, `csv_loader`,
+  `flax_loader`, `image_folder_loader`, `json_loader`, `system_loader`,
+  `tf_loader`) and `data.loaders.media_utils`, and the redundant
+  `dagnam/resources/datasets_upload.py` module. These were internal forwarders
+  with no external consumers; the canonical modules (`dagnam.resources.*`,
+  `dagnam._core.{client.common,resolver,sse}`, `dagnam.data.loaders.{audio,csv,
+  flax,image_folder,json_array,system,tf,media}`) are unchanged, and the
+  documented public API (`dagnam.*` exports, the `dagnam.resources.datasets_upload`
+  alias) keeps resolving.
+
+## [0.1.0] - 2026-05-10
+
+First public PyPI release of the official Dagnam.AI Python SDK.
+
+### Added
+
+- Dataset loading with API-key authentication, metadata lookup, local caching,
+  SHA-256 verification, LRU eviction, resumable downloads, presigned download
+  URLs, and dataset version selection.
+- `DagnamDataset` adapters for polars, PyTorch, TensorFlow, and Flax/JAX.
+- Tabular CSV, TSV, JSON, and JSONL loaders with deterministic train/validation
+  and test splits.
+- Image-folder and audio-folder dataset loaders with archive extraction,
+  deterministic fallback splits, and framework transform hooks.
+- System dataset resolution by friendly name.
+- Dataset upload helpers for local files and server-side URL ingestion.
+- Inference helpers for single and batch prediction plus deployment health.
+- Checkpoint download with checksum verification and a dedicated checkpoint
+  cache.
+- Synchronous SSE training stream iterator with reconnect support.
+- Deployment, Model Hub, project, and code generation resource modules.
+- `LongRunningOperation` for polling asynchronous platform operations.
+- `dagnam.aio.AsyncDagnamClient` for low-level async API access.
+- CLI commands for login, datasets, cache, inference, checkpoints, streams,
+  deployments, hub, projects, and code generation.
+- Typed package marker via `dagnam/py.typed`.
+
+### Security
+
+- `dagnam login` writes `~/.dagnam/config.json` with owner-only permissions on
+  POSIX systems and creates `~/.dagnam` with owner-only directory permissions.
+- Archive extraction rejects unsafe paths, symlinks, special files, and oversized
+  archives before unpacking media datasets.
+- Downloaded dataset and checkpoint filenames from `Content-Disposition` are
+  sanitized before writing to disk.
+
+### Changed
+
+- Licensed the SDK under Apache License 2.0 with a root `NOTICE` file included
+  in source and wheel distributions.
+
+### Compatibility
+
+- Python `>=3.12,<3.13`.
+- Dagnam backend `>=0.5.0, <0.7.0`.

dagnam-0.7.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,126 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+- Demonstrating empathy and kindness toward other people
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+- Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+- The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+- Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+`dagnam@dagnam.ai`. All complaints will be reviewed and investigated promptly
+and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact:** Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence:** A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact:** A violation through a single incident or series of
+actions.
+
+**Consequence:** A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. Violating
+these terms may lead to a temporary or permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact:** A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence:** A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. Violating these
+terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact:** Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence:** A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][mozilla].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[mozilla]: https://github.com/mozilla/diversity
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][faq]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[faq]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations