cognite-toolkit 0.6.21__tar.gz → 0.6.87__tar.gz

cognite_toolkit-0.6.87/.gemini/styleguide.md

@@ -0,0 +1,176 @@
+# Cognite Python Style Guide
+
+## Key Principles
+
+- **Strong Typing**: Use type hints extensively with pyright. Avoid `Any` when possible
+- **Type Safety**: Use dataclasses and Pydantic models for complex data
+  structures instead of untyped dictionaries
+- **IO Safety**: Always use typed data structures for file operations and data parsing
+- **Readability**: Code should be immediately understandable
+- **Maintainability**: Write code that is easy to modify and extend
+- **Consistency**: Follow established patterns across the codebase
+
+## Principles on doing pull request reviews
+
+- **Main point first.** Start with the key feedback or required action.
+- **Be concise.** Use short, direct comments. Avoid unnecessary explanations.
+- **Actionable suggestions.** If something needs fixing, state exactly what and how.
+- **One issue per comment.** Separate unrelated feedback for clarity.
+- **Code, not prose.** Prefer code snippets or examples over long text.
+- **Background only if needed.** Add context only if the main point isn't obvious.
+
+## How to do pull request summaries
+
+- **Short recap.** Summarize the main point of the PR in one or two sentences.
+- **Don't repeat the PR description.** Only add new or clarifying information.
+- **Be brief unless needed.** Only write a longer summary if the PR description
+  is missing crucial details.
+- **Extend, don't duplicate.** If more detail is needed, clearly state what is
+  missing from the PR description and add only the necessary context.
+
+## Line Length and Formatting
+
+- **Maximum line length**: 120 characters (configured in ruff)
+- **Target Python version**: 3.10+
+- **Indentation**: 4 spaces per level
+
+## Type Hints
+
+- **Required**: All functions, methods, and class attributes must have type hints
+- **Avoid `Any`**: Use specific types whenever possible
+- **Complex data**: Use dataclasses or Pydantic models instead of `dict[str, Any]`
+- **File operations**: Always parse file content into typed structures
+
+```python
+# Good - typed data structure
+@dataclass
+class Config:
+    model_name: str
+    temperature: float
+    max_tokens: int
+
+def load_config(path: Path) -> Config:
+    data = json.loads(path.read_text())
+    return Config(**data)
+
+# Bad - untyped dictionary
+def load_config(path: Path) -> dict[str, Any]:
+    return json.loads(path.read_text())
+```
+
+## Imports
+
+- **Group imports**: Standard library, third-party, local application
+- **Absolute imports**: Always use absolute imports for clarity
+- **Sort alphabetically** within groups
+- **Type checking imports**: Use `TYPE_CHECKING` for type-only imports
+
+```python
+import json
+import logging
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+from cognite.client import CogniteClient
+from pydantic import BaseModel
+
+from cog_ai.common.types import ModelConfig
+
+if TYPE_CHECKING:
+    from cog_ai.tools.query.types import QueryCompletion
+```
+
+## Naming Conventions
+
+- **Variables/functions**: `snake_case`
+- **Constants**: `UPPER_SNAKE_CASE`
+- **Classes**: `PascalCase`
+- **Modules**: `snake_case`
+- **Private members**: Single leading underscore `_private`
+
+## Docstrings
+
+Use concise docstrings with Args/Returns format. Based on repository patterns:
+
+```python
+def render_header(header: str) -> str:
+    """
+    Renders a (markdown) heading.
+
+    Args:
+        header (str): header
+
+    Returns:
+        str: The rendered header
+    """
+    return f"{header}\n{'=' * len(header)}\n"
+
+def walk_sdk_documentation(content: Tag, parser: Parser[T]) -> Iterable[T]:
+    """Parse the content of a file and yields documents. The parser controls how
+    the sections are transformed into documents."""
+    # Implementation here
+```
+
+**Docstring patterns**:
+
+- Start with a concise description
+- Use `Args:` and `Returns:` for complex functions
+- Omit obvious parameter descriptions
+- Keep descriptions brief and factual
+- Ok for `__init__` methods to omit docstring
+
+## Error Handling
+
+- **Specific exceptions**: Avoid broad `Exception` catches
+- **Graceful handling**: Provide meaningful error messages
+- **Type safety**: Return `None` or use Union types for fallible operations
+
+```python
+def load_llm_response(response: str) -> dict[str, Any] | None:
+    try:
+        return json.loads(response)
+    except (TypeError, json.JSONDecodeError) as e:
+        log.warning(f"Failed json load response from LLM: {e}")
+        return None
+```
+
+## Tooling
+
+- **Formatter**: `uv run ruff format --force-exclude --quiet`
+- **Linter**: `uv run ruff check --force-exclude --fix --exit-non-zero-on-fix`
+- **Type checker**: `dmypy run -- cognite_toolkit/ --config-file pyproject.toml` (uses mypy)
+- **Pre-commit**: `pre-commit run --all-files` for comprehensive checks
+
+## Ruff Configuration Deviations
+
+Current project configuration deviates from PEP 8:
+
+- **Line length**: 120 characters (vs PEP 8's 79)
+- **Ignored rules**: E501 (line too long), UP017 (datetime.timezone.utc)
+- **Selected rules**: Includes bugbear (B0), pyupgrade (UP), and isort (I)
+
+## Data Structures
+
+**Prefer typed structures**:
+
+```python
+# Good
+@dataclass
+class FunctionError:
+    function_name: str
+    message: str
+
+# Good
+class QueryCompletion(BaseModel):
+    query: str
+    variables: dict[str, Any]
+
+# Avoid
+error_data = {"function_name": "foo", "message": "bar"}
+```
+
+## Logging
+
+- Use the `logging` module with appropriate levels
+- Include contextual information for debugging
+- Format: `log.warning(f"Description: {variable}")`

cognite_toolkit-0.6.87/.github/CODEOWNERS

@@ -0,0 +1,3 @@
+* @cognitedata/partner-enablement @cognitedata/velocity
+/cognite_toolkit/cognite_modules/core/cdf_apm_base/* @cognitedata/partner-enablement @cognitedata/opsup-team-dev
+/cognite_toolkit/cognite_modules/infield/* @cognitedata/partner-enablement @cognitedata/opsup-team-dev

cognite_toolkit-0.6.87/.github/workflows/demo.yml

@@ -0,0 +1,44 @@
+name: Deploy modules to demo project
+on:
+  push:
+    branches:
+      - main
+env:
+  PYTHON_VERSION: '3.10'
+
+
+jobs:
+  build-modules:
+    runs-on: ubuntu-latest
+    env:
+        CDF_CLUSTER: ${{ vars.CDF_CLUSTER }}
+        CDF_PROJECT: ${{ vars.CDF_PROJECT }}
+        IDP_CLIENT_ID: ${{ vars.IDP_CLIENT_ID }}
+        IDP_CLIENT_SECRET: ${{ secrets.IDP_CLIENT_SECRET }}
+        IDP_TOKEN_URL: ${{ vars.IDP_TOKEN_URL }}
+        LOGIN_FLOW: client_credentials
+    name: Loadmaster
+    steps:
+      - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5
+      - uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+      - name: Install dependencies
+        run: |
+          pip install uv
+          uv sync --all-extras
+      - name: Build the package
+        run: uv build
+      - name: Initialize project
+        run: uv run cdf modules init demo_project --clean --all --library-url "https://github.com/cognitedata/toolkit-data/releases/download/latest/packages.zip" --library-checksum "sha256:b70c207a78d6c94e0977a733571420b250ca641ac8802d8310f1ffe05bf7b2eb"
+      - name: "Pre-processing for demo environment"
+        run: uv run python ./demo/preproc.py
+      - name: "Not collect metrics"
+        run: uv run cdf collect opt-out
+      - name: "Build the templates"
+        run: uv run cdf build --env demo -o ./demo_project
+      - name: "Verify and create access rights"
+        run: uv run cdf auth verify --no-prompt
+      - name: "Deploy the templates"
+        run: |
+          uv run cdf deploy --env demo

cognite_toolkit-0.6.87/.github/workflows/release.yaml

@@ -0,0 +1,103 @@
+name: Release Toolkit
+on:
+  push:
+    branches:
+      - main
+env:
+  PYTHON_VERSION: '3.11'
+
+jobs:
+  upload_coverage:
+    name: Upload coverage report
+    runs-on: ubuntu-latest
+    permissions: read-all
+    env:
+      CDF_CLUSTER: ${{ vars.CDF_CLUSTER }}
+      CDF_PROJECT: ${{ vars.CDF_PROJECT }}
+      IDP_CLIENT_ID: ${{ vars.IDP_CLIENT_ID }}
+      IDP_CLIENT_SECRET: ${{ secrets.IDP_CLIENT_SECRET }}
+      IDP_TOKEN_URL: ${{ vars.IDP_TOKEN_URL }}
+    steps:
+      - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5
+      - uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6
+        with:
+          python-version: 3.11
+      - name: Install dependencies
+        run: |
+          pip install uv
+          uv sync --all-extras
+      - name: Create test coverage report
+        env:
+          IS_GITHUB_ACTIONS: "true"
+        run: uv run pytest --cov=cognite_toolkit/ --cov-config=pyproject.toml --cov-report=xml:coverage.xml tests/ -n8
+      - uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          slug: cognitedata/toolkit
+
+  release-pypi-docker-hub:
+    runs-on: ubuntu-latest
+    environment: CD
+    steps:
+      - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+      - name: Install dependencies
+        run: |
+          pip install uv
+          uv sync --all-extras
+      - name: Dump last commit message to file
+        run: git log -1 --pretty=%B > last_git_message.txt
+      - name: Dump last tag to file
+        run: git describe --tags --abbrev=0 > last_version.txt
+      - name: Bump Version
+        run: uv run python dev.py bump --verbose
+      - id: version
+        name: Read the new version
+        # Read the version from the cognite_toolkit/_version.py file
+        run: echo "version=$(sed -n 's/^__version__ = "\(.*\)"/\1/p' cognite_toolkit/_version.py)" >> $GITHUB_ENV
+      - name: Create CHANGELOG entry
+        if: env.version != '0.0.0'
+        run: uv run python dev.py changelog
+      - name: Build package
+        if: env.version != '0.0.0'
+        run: uv build
+      - name: Release to PyPI
+        if: env.version != '0.0.0'
+        env:
+          TWINE_USERNAME: __token__
+          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
+        run: uv run twine upload --skip-existing --verbose dist/*
+      - name: Add entry to CHANGELOG
+        if: env.version != '0.0.0'
+        uses: ncipollo/release-action@b7eabc95ff50cbeeedec83973935c8f306dfcd0b # v1
+        with:
+            token: ${{ secrets.GITHUB_TOKEN }}
+            prerelease: false
+            draft: false
+            tag: ${{ env.version }}
+            bodyFile: last_changelog_entry.md
+
+      - name: Login to Docker Hub
+        if: env.version != '0.0.0'
+        uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3
+        with:
+          username: ${{ secrets.DOCKERHUB_USER }}
+          password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}
+
+      - name: Set up Docker Buildx
+        if: env.version != '0.0.0'
+        uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3
+
+      - name: Build and push
+        if: env.version != '0.0.0'
+        uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6
+        with:
+          context: .
+          file: ./Dockerfile
+          platforms: linux/amd64,linux/arm64/v8
+          push: true
+          tags: cognite/toolkit:${{ env.version }}, cognite/toolkit:latest

cognite_toolkit-0.6.87/.gitignore

@@ -0,0 +1,317 @@
+# File created using '.gitignore Generator' for Visual Studio Code: https://bit.ly/vscode-gig
+# Created by https://www.toptal.com/developers/gitignore/api/visualstudiocode,python,macos,git,windows
+# Edit at https://www.toptal.com/developers/gitignore?templates=visualstudiocode,python,macos,git,windows
+
+### Git ###
+# Created by git for backups. To disable backups in Git:
+# $ git config --global mergetool.keepBackup false
+*.orig
+
+# Created by git when using merge tools for conflicts
+*.BACKUP.*
+*.BASE.*
+*.LOCAL.*
+*.REMOTE.*
+*_BACKUP_*.txt
+*_BASE_*.txt
+*_LOCAL_*.txt
+*_REMOTE_*.txt
+
+### macOS ###
+# General
+.DS_Store
+.AppleDouble
+.LSOverride
+
+# Icon must end with two \r
+Icon
+
+
+# Thumbnails
+._*
+
+# Files that might appear in the root of a volume
+.DocumentRevisions-V100
+.fseventsd
+.Spotlight-V100
+.TemporaryItems
+.Trashes
+.VolumeIcon.icns
+.com.apple.timemachine.donotpresent
+
+# Directories potentially created on remote AFP share
+.AppleDB
+.AppleDesktop
+Network Trash Folder
+Temporary Items
+.apdisk
+
+### macOS Patch ###
+# iCloud generated files
+*.icloud
+
+### Python ###
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+*.env
+.venv
+.local/
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+.idea/
+
+### Python Patch ###
+# Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration
+poetry.toml
+
+# ruff
+.ruff_cache/
+
+# LSP config files
+pyrightconfig.json
+
+### VisualStudioCode ###
+.vscode/*
+.vscode/settings.json
+!.vscode/tasks.json
+!.vscode/launch.json
+!.vscode/extensions.json
+!.vscode/*.code-snippets
+
+# Local History for Visual Studio Code
+.history/
+
+# Built Visual Studio Code Extensions
+*.vsix
+
+### VisualStudioCode Patch ###
+# Ignore all local history of files
+.history
+.ionide
+
+### Windows ###
+# Windows thumbnail cache files
+Thumbs.db
+Thumbs.db:encryptable
+ehthumbs.db
+ehthumbs_vista.db
+
+# Dump file
+*.stackdump
+
+# Folder config file
+[Dd]esktop.ini
+
+# Recycle Bin used on file shares
+$RECYCLE.BIN/
+
+# Windows Installer files
+*.cab
+*.msi
+*.msix
+*.msm
+*.msp
+
+# Windows shortcuts
+*.lnk
+
+# End of https://www.toptal.com/developers/gitignore/api/visualstudiocode,python,macos,git,windows
+
+# Custom rules (everything added below won't be overriden by 'Generate .gitignore File' if you use 'Update' option)
+
+tmp/
+playground/
+new_project/
+# If you need to update the cognite_toolkit template files for local.yaml and config.yaml, comment below
+local.yaml
+config.yaml
+!demo/config.yaml
+demo_project/
+tests/tests_unit/pytest-project/
+my_project/
+my_organization/
+*.bck
+
+# Hide environments used for migration testing
+**/.venv*
+module_upgrade/tmp*
+
+cognite_toolkit/config.local.yaml
+**/.venv*
+.venv.*
+build.*
+cognite_toolkit/.env.*
+cognite_toolkit/modules/*
+module_upgrade/project_inits
+function_local_venvs/
+tests/data/run_data/build_info.dev.yaml
+orgdir*
+module_upgrade/cdf.toml
+# Org used by bootcamp
+ice-cream-dataops
+module_upgrade/*.log
+needle/
+
+tmp.py
+tmp*.py
+my_*/
+function_local_venvs*/
+qa_pending_id/
+tests/**/cdf.toml
+
+*.xlsx
+data/
+migration_logs*/

cognite_toolkit-0.6.87/.markdownlint.yaml

@@ -0,0 +1,10 @@
+{  
+    "MD033": false,
+    "MD024": {
+        "siblings_only": true
+    },
+    "MD013": {
+        "line_length": 120,
+        "tables"    : false,
+    },
+}

cognite_toolkit-0.6.87/.pre-commit-config.yaml

@@ -0,0 +1,41 @@
+---
+repos:
+  - repo: https://github.com/astral-sh/uv-pre-commit
+    # uv version.
+    rev: 0.9.7
+    hooks:
+      - id: uv-lock
+
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    hooks:
+      - id: ruff-check
+        args:
+          - --fix
+          - --exit-non-zero-on-fix
+          - --ignore=E731,E501,W605,T201,UP007
+            # See https://beta.ruff.rs/docs/rules for an overview of ruff rules
+          - --select=E,W,F,I,T,RUF,TID,UP
+          - --fixable=E,W,F,I,T,RUF,TID,UP
+          - --target-version=py310
+      - id: ruff-format
+    rev: v0.14.3
+
+  - repo: https://github.com/igorshubovych/markdownlint-cli
+    rev: v0.45.0
+    hooks:
+    - id: markdownlint
+    
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v6.0.0 # Use the latest stable version
+    hooks:
+      - id: end-of-file-fixer
+
+# Mypy must be run in the local system environment, not in the pre-commit environment.
+  - hooks:
+    - id: mypy
+      name: mypy
+      entry: dmypy run -- cognite_toolkit/ --config-file pyproject.toml
+      files: ^.*.(py|pyi)$
+      language: system
+      pass_filenames: false
+    repo: local