govio 0.2.11__tar.gz

govio-0.2.11/.gitignore

@@ -0,0 +1,23 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+.trae/
+
+# Virtual environments
+.venv
+logs/
+data/
+.env
+uv.lock
+
+assets/
+.iflow
+.worktrees/
+
+# observe DataFrame storage
+.govio/observe/
+.claude/settings.local.json

govio-0.2.11/AGENTS.md

@@ -0,0 +1,251 @@
+# AGENTS.md
+
+Guidelines for AI coding agents working in this repository.
+
+## Project Overview
+
+Govio is a data governance knowledge graph library built on NetworkX. It provides metadata management, data standard recommendation, and graph database integration capabilities.
+
+- **Language**: Python 3.13+
+- **Package Manager**: uv
+- **Build Backend**: hatchling
+
+## Build/Lint/Test Commands
+
+### Installation
+
+```bash
+# Install dependencies
+uv sync
+
+# Install with dev dependencies
+uv sync --group dev
+```
+
+### Running Tests
+
+```bash
+# Run all tests
+uv run pytest tests/
+
+# Run a single test file
+uv run pytest tests/test_relationship.py
+
+# Run a specific test
+uv run pytest tests/test_relationship.py::test_load_json_success
+
+# Run tests with verbose output
+uv run pytest tests/ -v
+
+# Run tests with coverage
+uv run pytest tests/ --cov=src/govio
+```
+
+### Linting and Formatting
+
+```bash
+# Check code with ruff
+uv run ruff check src/ tests/
+
+# Format code with ruff
+uv run ruff format src/ tests/
+
+# Fix linting issues automatically
+uv run ruff check --fix src/ tests/
+```
+
+### Build
+
+```bash
+# Build package
+uv build
+
+# Build wheel only
+uv build --wheel
+```
+
+### Type Checking
+
+```bash
+# Run pyright (if available)
+uv run pyright src/
+```
+
+## Code Style Guidelines
+
+### Imports
+
+Order imports in three groups, separated by blank lines:
+
+1. Standard library imports (alphabetical)
+2. Third-party imports (alphabetical)
+3. Local imports (alphabetical)
+
+```python
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+import pandas as pd
+from sklearn.feature_extraction.text import TfidfVectorizer
+
+from .application import AppInfoLoader
+from .database import DatabaseLoader
+```
+
+### Type Hints
+
+Use modern Python 3.13+ type hint syntax:
+
+```python
+# Preferred
+def load_tables(self, schema_limits: list[str] | None) -> pd.DataFrame:
+    ...
+
+# Avoid (old style)
+from typing import List, Optional
+def load_tables(self, schema_limits: Optional[List[str]]) -> pd.DataFrame:
+    ...
+```
+
+Use union types with `|` for optional parameters. Use `Any` sparingly.
+
+### Naming Conventions
+
+- **Modules**: snake_case (`recommender.py`, `relationship.py`)
+- **Classes**: PascalCase (`StandardRecommender`, `RelationshipLoader`)
+- **Functions/Methods**: snake_case (`load_relationships`, `find_k_neighbors`)
+- **Private methods**: prefix with underscore (`_preprocess_std_data`, `_validate_inputs`)
+- **Constants**: UPPER_SNAKE_CASE (`DEFAULT_WEIGHTS`, `MIN_SIMILARITY`)
+- **Properties**: snake_case with `@property` decorator (`PhysicalTable`, `Col`)
+
+### Docstrings
+
+Use Chinese docstrings for Chinese-language projects. Include Args and Returns sections:
+
+```python
+def validate_relationship(self, rel: dict[str, Any], index: int) -> bool:
+    """验证单个关系的有效性
+
+    Args:
+        rel: 关系字典
+        index: 关系索引（用于错误消息）
+
+    Returns:
+        bool: 是否有效
+    """
+```
+
+### Classes
+
+- Use `__init__` for initialization with type-annotated parameters
+- Use `@property` for computed attributes that don't require parameters
+- Use factory functions for complex object creation
+
+```python
+class DatabaseLoader:
+    def __init__(self, db: str, workspace_uuid: str, schema_limits: list[str] | None = None) -> None:
+        self.engine = create_engine(db)
+        self.workspace_uuid = workspace_uuid
+
+    @property
+    def PhysicalTable(self) -> pd.DataFrame:
+        return self.load_tables()
+```
+
+### Error Handling
+
+- Raise descriptive exceptions with context
+- Use logging module for warnings (not print statements)
+- Validate inputs early in public methods
+
+```python
+def _validate_inputs(self):
+    if not self.json_path.exists():
+        raise FileNotFoundError(f"关系文件不存在: {self.json_path}")
+
+    if self.df_tables.empty:
+        raise ValueError("PhysicalTable DataFrame 为空")
+```
+
+### Comments
+
+- Write comments in Chinese for this codebase
+- Avoid inline comments that restate the obvious
+- Use module-level docstrings to explain purpose
+
+### File Organization
+
+```
+src/govio/
+├── __init__.py          # Public exports only
+├── graph/               # Graph database implementations
+│   ├── __init__.py
+│   ├── networkx_graph.py
+│   └── falkordb_graph.py
+└── metadata/            # Metadata loading and processing
+    ├── __init__.py
+    ├── database.py
+    ├── recommender.py
+    └── relationship.py
+```
+
+### Constants and Configuration
+
+Define module-level constants at the top of the file:
+
+```python
+DEFAULT_WEIGHTS = {
+    'table': 0.20,
+    'name': 0.26,
+    'comment': 0.22,
+    'type': 0.22,
+    'numeric': 0.10
+}
+
+DEFAULT_K_NEIGHBORS = 5
+MIN_SIMILARITY = 0.7
+```
+
+### Avoid
+
+- Adding comments that restate code
+- Using `print()` for logging (use `logging` module)
+- Mutable default arguments
+- Bare `except` clauses
+- Star imports (`from module import *`)
+
+## Project-Specific Notes
+
+### Entry Points
+
+The package defines CLI entry points in `pyproject.toml`:
+
+- `metadata` - Generate metadata CSV files
+- `gml_generate` - Generate GML graph files
+
+### Environment Variables
+
+Load environment variables using `python-dotenv`:
+
+```python
+from dotenv import load_dotenv
+import os
+
+load_dotenv()
+db = os.getenv("KUNDB_URL", "")
+```
+
+### Testing
+
+Tests use pytest with fixtures. Place fixtures at module level or in conftest.py:
+
+```python
+@pytest.fixture
+def sample_tables():
+    return pd.DataFrame({
+        "full_table_name": ["db.schema.table1"],
+    })
+```

govio-0.2.11/CLAUDE.md

@@ -0,0 +1,71 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Govio is a Python data governance knowledge graph platform. It extracts metadata from MySQL databases, builds graph structures (via FalkorDB or NetworkX), and provides data standard recommendation using collaborative filtering (k-NN). The name combines "Governance" + "IO" (data interaction/flow).
+
+## Commands
+
+```bash
+# Install dependencies (uses uv package manager, Tsinghua mirror)
+uv sync
+uv sync --group dev        # includes falkordb-bulk-loader
+
+# Run tests (pytest-style tests in tests/, but using unittest runner in README)
+uv run pytest tests/
+
+# CLI entry points (defined in pyproject.toml [project.scripts])
+metadata --kundb "mysql+pymysql://user:pass@host/db" --app-list "path.xlsx" --app-map "path.json" -o ./output
+metadata --kundb "..." --app-list "..." --app-map "..." -m recommend -o ./output
+metadata --kundb "..." --app-list "..." --app-map "..." --relationship rel.json -o ./output
+gml_generate --csv ./output -o ./output
+```
+
+## Architecture
+
+### Source layout: `src/govio/`
+
+**`__init__.py`** — Public API surface: exports `run`, `gml_generate`, `FalkorDBGraph`, `NetworkXGraph`.
+
+**`graph/`** — Graph database abstractions:
+- `networkx_graph.py` — In-memory graph via NetworkX GML files. Provides `schema` property (node/edge type inspection) and direct `G` access to the underlying `nx.DiGraph`.
+- `falkordb_graph.py` — FalkorDB (Redis-based) graph client using Cypher queries.
+
+**`metadata/`** — Data loading and processing pipeline:
+- `database.py` — `DatabaseLoader`: extracts table/column metadata from MySQL via SQLAlchemy. Exposes `PhysicalTable` and `Col` as DataFrames. Requires `workspace_uuid` and `schema_limits`.
+- `application.py` — `AppInfoLoader`: loads app metadata from Excel (openpyxl).
+- `standard.py` — `StandardLoader`: loads data standards and compliance info from governance DB.
+- `relationship.py` — `RelationshipLoader` / `load_relationships()`: validates table relationships from JSON (supports one_to_one, one_to_many, many_to_one, many_to_many). Returns edges DataFrame.
+- `recommender.py` — `create_recommender()`: k-NN collaborative filtering for recommending data standards to non-compliant columns. Uses configurable weights (table, name, comment, type, numeric).
+- `metric.py` — `MetricLoader` / `load_metrics()`: loads metric definitions from JSON (validated by `metric_schema.json`), produces Metric/Dimension node DataFrames and edge DataFrames (USES_TABLE, REFERS_COLUMN, DERIVED_FROM, DIMENSION_USED, SUPERSEDES). Validates source table references, derived_from references, and DAG property.
+- `gen_networkx.py` — `build_graph()`: converts CSV node/edge files to NetworkX GML format. Reads specific CSV naming conventions (`:ID(NodeType)` columns for nodes, `:START_ID`/`:END_ID` for edges).
+- `utility.py` — CLI entry point (`run()`), orchestrates the full pipeline: load metadata → generate CSVs → optionally produce GML. Also contains `data_standard_recommend()` for batch recommendation mode.
+
+### Graph model
+
+Node types: `PhysicalTable`, `Col`, `Application`, `Standard`, `Metric`, `Dimension`
+Edge types: `HAS_COLUMN` (table→col), `USE` (app→table), `COMPLIES_WITH` (col→standard), `RELATES_TO` (table→table), `USES_TABLE` (metric→table), `REFERS_COLUMN` (metric→col), `DERIVED_FROM` (metric→metric), `DIMENSION_USED` (metric→dimension), `SUPERSEDES` (metric→metric)
+
+`Calculation` node type and `CALCULATED_BY`/`BASED_ON` edges are reserved for future shared calculation templates.
+
+CSV files use FalkorDB bulk-import header conventions (`:ID(Type)`, `:START_ID(Type)`, `:END_ID(Type)`). The GML generator parses these headers to reconstruct typed graphs.
+
+### Data flow
+
+1. `metadata` CLI → DatabaseLoader + AppInfoLoader + StandardLoader → CSV files (node + edge)
+2. `metadata --relationship` appends RELATES_TO.csv
+3. `metadata` with `--metric` (via onboard) appends Metric.csv, Dimension.csv, and metric edge CSVs
+4. `metadata -m recommend` generates COMPLIES_WITH.csv via recommender
+4. `gml_generate` → CSV files → NetworkX GML graph
+5. `NetworkXGraph` loads GML for query/inspection
+
+### Key conventions
+
+- Python 3.13+, uses modern type hints (`X | None` syntax, not `Optional[X]`)
+- All metadata loaders return pandas DataFrames
+- Node identities use dotted format: `db.schema.table.column`
+- `workspace_uuid` is hardcoded in utility.py but parameterized in loader classes
+- Tests use pytest (despite README mentioning unittest) — run with `uv run pytest`
+- Chinese language is used in comments, print statements, and documentation

govio-0.2.11/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 chenxinma
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.