datavane 0.1.0__tar.gz

datavane-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Antonio Jesús Salido Ranea
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

datavane-0.1.0/PKG-INFO

@@ -0,0 +1,166 @@
+Metadata-Version: 2.4
+Name: datavane
+Version: 0.1.0
+Summary: A Python library for automatic dataset documentation
+License-Expression: MIT
+License-File: LICENSE
+Author: Antonio Jesús Salido Ranea
+Requires-Python: >=3.11
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Provides-Extra: dev
+Requires-Dist: pytest ; extra == "dev"
+Requires-Dist: ruff ; extra == "dev"
+Description-Content-Type: text/markdown
+
+# datavane
+
+A Python library for automatically generating documentation from datasets.
+
+`datavane` inspects JSON and CSV datasets, infers their structure and generates a Markdown data dictionary containing information about fields, types, examples, presence and null values.
+
+## Features
+
+* Support for JSON and CSV files.
+* Recursive processing of nested dictionaries and lists.
+* Automatic field type inference.
+* Field path notation for nested structures.
+* Example values for detected fields.
+* Field presence and null-value statistics.
+* Recursive discovery of JSON and CSV files inside directories.
+* Markdown documentation generation.
+* No external runtime dependencies.
+
+## Installation
+
+Install Datavane from PyPI:
+
+```bash
+pip install datavane
+```
+
+Once installed, you can import it directly in your Python project:
+
+```python
+from datavane import doc_table
+```
+
+
+## Usage
+
+The main public API is `doc_table()`:
+
+```python
+from pathlib import Path
+
+from data_dictionary import doc_table
+
+doc_table(
+    Path("data/matches.json"),
+    "Matches",
+    Path("docs/matches.md"),
+)
+```
+
+The function accepts either a JSON/CSV file or a directory containing supported datasets.
+
+For example, given a dataset containing:
+
+```json
+{
+    "id": 1,
+    "name": "Juan",
+    "address": {
+        "city": "Málaga"
+    }
+}
+```
+
+the generated documentation can identify fields such as:
+
+```text
+id
+name
+address
+address.city
+```
+
+along with their inferred types, examples and statistics.
+
+## Generated documentation
+
+The generated Markdown document contains a summary followed by the inferred schema:
+
+```markdown
+# Matches
+
+> Data dictionary generated automatically by `datavane`.
+
+## Dataset summary
+
+| Property | Value |
+|----------|-------|
+| Records  | 100   |
+| Fields   | 8     |
+
+## Schema
+
+| Field | Type | Example | Presence | Nulls |
+|---|---|---|---|---|
+| id | int | 1 | 100.0% | 0.0% |
+| name | str | John | 100.0% | 0.0% |
+| address.city | str | Madrid | 98.0% | 0.0% |
+```
+
+## Project structure
+
+```text
+data-dictionary/
+├── src/
+│   └── data_dictionary/
+│       ├── __init__.py
+│       ├── models.py
+│       ├── inference.py
+│       ├── formatting.py
+│       ├── io.py
+│       └── generator.py
+├── tests/
+│   ├── fixtures/
+│   ├── unit/
+│   └── integration/
+├── pyproject.toml
+├── README.md
+└── LICENSE
+```
+
+## Development
+
+Create a virtual environment and install the project in editable mode:
+
+```bash
+python -m venv .venv
+python -m pip install -e .
+```
+
+Run the test suite with:
+
+```bash
+pytest
+```
+
+The project contains both unit and integration tests.
+
+## Status
+
+This project is currently in early development. The public API and internal implementation may change as the library evolves.
+
+## License
+
+This project is licensed under the MIT License.
+

datavane-0.1.0/README.md

@@ -0,0 +1,144 @@
+# datavane
+
+A Python library for automatically generating documentation from datasets.
+
+`datavane` inspects JSON and CSV datasets, infers their structure and generates a Markdown data dictionary containing information about fields, types, examples, presence and null values.
+
+## Features
+
+* Support for JSON and CSV files.
+* Recursive processing of nested dictionaries and lists.
+* Automatic field type inference.
+* Field path notation for nested structures.
+* Example values for detected fields.
+* Field presence and null-value statistics.
+* Recursive discovery of JSON and CSV files inside directories.
+* Markdown documentation generation.
+* No external runtime dependencies.
+
+## Installation
+
+Install Datavane from PyPI:
+
+```bash
+pip install datavane
+```
+
+Once installed, you can import it directly in your Python project:
+
+```python
+from datavane import doc_table
+```
+
+
+## Usage
+
+The main public API is `doc_table()`:
+
+```python
+from pathlib import Path
+
+from data_dictionary import doc_table
+
+doc_table(
+    Path("data/matches.json"),
+    "Matches",
+    Path("docs/matches.md"),
+)
+```
+
+The function accepts either a JSON/CSV file or a directory containing supported datasets.
+
+For example, given a dataset containing:
+
+```json
+{
+    "id": 1,
+    "name": "Juan",
+    "address": {
+        "city": "Málaga"
+    }
+}
+```
+
+the generated documentation can identify fields such as:
+
+```text
+id
+name
+address
+address.city
+```
+
+along with their inferred types, examples and statistics.
+
+## Generated documentation
+
+The generated Markdown document contains a summary followed by the inferred schema:
+
+```markdown
+# Matches
+
+> Data dictionary generated automatically by `datavane`.
+
+## Dataset summary
+
+| Property | Value |
+|----------|-------|
+| Records  | 100   |
+| Fields   | 8     |
+
+## Schema
+
+| Field | Type | Example | Presence | Nulls |
+|---|---|---|---|---|
+| id | int | 1 | 100.0% | 0.0% |
+| name | str | John | 100.0% | 0.0% |
+| address.city | str | Madrid | 98.0% | 0.0% |
+```
+
+## Project structure
+
+```text
+data-dictionary/
+├── src/
+│   └── data_dictionary/
+│       ├── __init__.py
+│       ├── models.py
+│       ├── inference.py
+│       ├── formatting.py
+│       ├── io.py
+│       └── generator.py
+├── tests/
+│   ├── fixtures/
+│   ├── unit/
+│   └── integration/
+├── pyproject.toml
+├── README.md
+└── LICENSE
+```
+
+## Development
+
+Create a virtual environment and install the project in editable mode:
+
+```bash
+python -m venv .venv
+python -m pip install -e .
+```
+
+Run the test suite with:
+
+```bash
+pytest
+```
+
+The project contains both unit and integration tests.
+
+## Status
+
+This project is currently in early development. The public API and internal implementation may change as the library evolves.
+
+## License
+
+This project is licensed under the MIT License.

datavane-0.1.0/pyproject.toml

@@ -0,0 +1,49 @@
+[build-system]
+requires = ["poetry-core>=2.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[project]
+name = "datavane"
+version = "0.1.0"
+description = "A Python library for automatic dataset documentation"
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+authors = [
+    {name = "Antonio Jesús Salido Ranea"}
+]
+dependencies = []
+
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest",
+    "ruff",
+]
+
+[tool.ruff]
+line-length = 88
+target-version = "py311"
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "F",
+    "I",
+    "UP",
+]
+
+[tool.poetry]
+packages = [
+    { include = "datavane", from = "src" }
+]

datavane-0.1.0/src/datavane/__init__.py

@@ -0,0 +1,5 @@
+from .generator import doc_table
+
+__all__ = [
+    "doc_table",
+]

datavane-0.1.0/src/datavane/formatting.py

@@ -0,0 +1,155 @@
+from .models import FieldInfo
+
+MAX_DICT_FIELDS = 5
+MAX_LIST_FIELDS = 3
+MAX_STRING_LENGTH = 80
+
+
+def format_types(types: set[str]) -> str:
+    """Format a set of types as a sorted, comma-separated string.
+
+    Args:
+        types (set[str]): Set containing the inferred types of a field.
+
+    Returns:
+        str: A comma-separated string containing the types in sorted order.
+    """
+
+    return ", ".join(sorted(types))
+
+
+def format_presence(field: FieldInfo, total_records: int) -> str:
+    """Calculate and format the percentage of records containing a field.
+
+    Args:
+        field (FieldInfo): Field information containing the number of appearances.
+        total_records (int): Total number of records in the dataset.
+
+    Returns:
+        str: The field presence as a percentage string.
+    """
+
+    return str(round((field.appearances / total_records) * 100, 1)) + "%"
+
+
+def format_nulls(field: FieldInfo, total_records: int) -> str:
+    """Calculate and format the percentage of null values for a field.
+
+    Args:
+        field (FieldInfo): Field information containing the number of null values.
+        total_records (int): Total number of records in the dataset.
+
+    Returns:
+        str: The null value percentage as a string.
+    """
+
+    return str(round((field.null_count / total_records) * 100, 1)) + "%"
+
+
+def format_example(example) -> str:
+    """Format an example value for inclusion in the Markdown output.
+
+    Strings longer than ``MAX_STRING_LENGTH`` are truncated. Dictionaries
+    and lists are recursively formatted and limited to the configured
+    number of elements.
+
+    Args:
+        example: Example value to format.
+
+    Returns:
+        str: A string representation suitable for the generated documentation.
+    """
+
+    if example is None:
+        return "null"
+
+    if isinstance(example, str):
+        if len(example) > MAX_STRING_LENGTH:
+            return example[:MAX_STRING_LENGTH] + "..."
+        return example
+
+    if isinstance(example, dict):
+        examples_list = []
+
+        for i, (key, value) in enumerate(example.items()):
+            if i >= MAX_DICT_FIELDS:
+                examples_list.append("...")
+                break
+
+            examples_list.append(f"{key}: {format_example(value)}")
+
+        return "{ " + ", ".join(examples_list) + " }"
+
+    if isinstance(example, list):
+        examples_list = []
+
+        for i, entry in enumerate(example):
+            if i >= MAX_LIST_FIELDS:
+                examples_list.append("...")
+                break
+
+            examples_list.append(format_example(entry))
+
+        return "[ " + ", ".join(examples_list) + " ]"
+
+    return str(example)
+
+
+def format_summary(
+    table_name: str, total_fields: int, total_records: int, file
+) -> None:
+    """Write the dataset summary section to a file.
+
+    Args:
+        table_name (str): Name of the dataset or table being documented.
+        total_fields (int): Number of fields detected in the dataset.
+        total_records (int): Number of records in the dataset.
+        file: File-like object where the Markdown output is written.
+    """
+
+    file.write(f"# {table_name} \n")
+    file.write("> Data dictionary generated automatically by `datavane` \n")
+    file.write("## Dataset summary \n")
+    file.write("| Property | Value | \n")
+    file.write("|----------|-------| \n")
+    file.write(f"| Records | {total_records}| \n")
+    file.write(f"| Fields  | {total_fields} | \n")
+
+
+def markdown_row(field: FieldInfo, total_records: int) -> str:
+    """Format a field as a Markdown table row.
+
+    Args:
+        field (FieldInfo): Field information to format.
+        total_records (int): Total number of records in the dataset.
+
+    Returns:
+        str: A Markdown table row containing the field's type, example, presence,
+          and null percentage.
+    """
+
+    return (
+        f"| {field.name} | "
+        f"{format_types(field.types)} | "
+        f"{format_example(field.example)} | "
+        f"{format_presence(field, total_records)} | "
+        f"{format_nulls(field, total_records)} |\n"
+    )
+
+
+def markdown_table(fields_dict: dict[str, FieldInfo], total_records: int, file) -> None:
+    """Write the dataset schema as a Markdown table.
+
+    Args:
+        fields_dict (dict[str, FieldInfo]): Mapping of field names
+        to their inferred information.
+        total_records (int): Total number of records in the dataset.
+        file: File-like object where the Markdown output is written.
+    """
+
+    file.write("## Schema \n")
+    file.write("| Field | Type | Example | Presence | Nulls | \n")
+    file.write("|-------|------|---------| -------- | ----- |\n")
+    file.writelines(
+        markdown_row(field, total_records) for field in fields_dict.values()
+    )

datavane-0.1.0/src/datavane/generator.py

@@ -0,0 +1,27 @@
+from pathlib import Path
+
+from .formatting import format_summary, markdown_table
+from .inference import infer_schema
+from .io import load_dataset
+
+
+def doc_table(path: Path, table_name: str, output_path: Path) -> None:
+    """Generate a Markdown data dictionary for a dataset.
+
+    The dataset is loaded from the given path, its schema is inferred,
+    and the resulting documentation is written to the specified output
+    file.
+
+    Args:
+        path (Path): Path to the input dataset or directory containing datasets.
+        table_name (str): Name used as the title of the generated documentation.
+        output_path (Path): Path where the Markdown documentation will be written.
+    """
+
+    data = load_dataset(path)
+    total_records = len(data)
+    schema = infer_schema(data)
+    total_fields = len(schema)
+    with open(output_path, "w", encoding="utf-8") as f:
+        format_summary(table_name, total_fields, total_records, f)
+        markdown_table(schema, total_records, f)

datavane-0.1.0/src/datavane/inference.py

@@ -0,0 +1,97 @@
+from .models import FieldInfo
+
+
+def infer_type(value) -> str:
+    """Infer the type of a value, including nested lists.
+
+    Lists are represented using the inferred types of their elements.
+    Empty lists are represented simply as ``list``.
+
+    Args:
+        value: Value whose type should be inferred.
+
+    Returns:
+        str: A string representing the inferred type.
+    """
+
+    if value is None:
+        return "null"
+
+    if isinstance(value, list):
+        if not value:
+            return "list"
+
+        element_types = set()
+
+        for element in value:
+            element_types.add(infer_type(element))
+
+        return f"list[{', '.join(sorted(element_types))}]"
+
+    return type(value).__name__
+
+
+def process_object(
+    obj: dict | list,
+    fields: dict[str, FieldInfo],
+    prefix: str = "",
+    context_count: int = 1,
+) -> None:
+    """Recursively process a dictionary or list to infer field information.
+
+    Nested fields are represented using dot notation, while fields inside
+    lists are marked with ``[]``. Field statistics are updated as values
+    are encountered.
+
+    Args:
+        obj (dict | list): Dictionary or list to process.
+        fields (dict[str, FieldInfo]): Mapping of field paths to their corresponding
+        ``FieldInfo``.
+        prefix (str, optional): Path prefix used for nested fields. Defaults to "".
+        context_count (int, optional): Number of records represented by the current
+        context. Defaults to 1.
+    """
+
+    if isinstance(obj, dict):
+        for field, value in obj.items():
+            if prefix:
+                field_name = f"{prefix}.{field}"
+            else:
+                field_name = field
+
+            if field_name not in fields:
+                fields[field_name] = FieldInfo(
+                    name=field_name, context_count=context_count
+                )
+
+            fields[field_name].update(value, infer_type(value))
+
+            if isinstance(value, (dict, list)):
+                process_object(value, fields, field_name, context_count=1)
+
+    elif isinstance(obj, list):
+        for entry in obj:
+            if isinstance(entry, dict):
+                process_object(entry, fields, prefix + "[]", context_count=len(obj))
+
+
+def infer_schema(data: list[dict]) -> dict[str, FieldInfo]:
+    """Infer the schema of a dataset.
+
+    Each record is recursively processed to collect information about
+    its fields, including their types, examples, appearances, nulls,
+    and contextual occurrence counts.
+
+    Args:
+        data (list[dict]): Dataset represented as a list of dictionaries.
+
+    Returns:
+        dict[str, FieldInfo]: A dictionary mapping field paths to their inferred
+        ``FieldInfo``.
+    """
+
+    fields: dict[str, FieldInfo] = {}
+    for entry in data:
+        process_object(entry, fields, context_count=len(data))
+
+    return dict(sorted(fields.items()))

datavane-0.1.0/src/datavane/io.py

@@ -0,0 +1,79 @@
+import json
+from csv import DictReader
+from pathlib import Path
+
+SUPPORTED_EXTENSIONS = {".json", ".csv"}
+
+
+def load_csv(file_path: Path) -> list[dict]:
+    """Load records from a CSV file.
+
+    The first row of the CSV file is used as the field names.
+
+    Args:
+        file_path (Path): Path to the CSV file.
+
+    Returns:
+        list[dict]: A list of dictionaries containing the CSV records.
+    """
+    with open(file_path, newline="", encoding="utf-8") as csvfile:
+        reader = DictReader(csvfile)
+        return list(reader)
+
+
+def load_json(file_path: Path) -> list[dict]:
+    """Load records from a JSON file.
+
+    Args:
+        file_path (Path): Path to the JSON file.
+
+    Returns:
+        list[dict]: The records contained in the JSON file.
+    """
+    with open(file_path, encoding="utf-8") as jsonfile:
+        data = json.load(jsonfile)
+
+    return data
+
+
+def load_dataset(path: Path) -> list[dict]:
+    """Load a dataset from a file or directory.
+
+    JSON and CSV files are supported. When a directory is provided,
+    supported files are searched recursively and their records are
+    combined into a single dataset.
+
+    Args:
+        path (Path): Path to a JSON/CSV file or a directory containing them.
+
+    Raises:
+        ValueError: If a file has an unsupported extension.
+        FileNotFoundError: If the provided path does not exist or is neither a file
+        nor a directory.
+
+    Returns:
+        list[dict]: A list of dictionaries containing the dataset records.
+    """
+
+    if path.is_file():
+        if path.suffix.lower() == ".json":
+            return load_json(path)
+        elif path.suffix.lower() == ".csv":
+            return load_csv(path)
+        else:
+            raise ValueError(
+                f"File path {path} not supported. File extension must be .csv or .json"
+            )
+    elif path.is_dir():
+        data = []
+        for file in path.rglob("*"):
+            if file.suffix.lower() not in SUPPORTED_EXTENSIONS:
+                continue
+
+            if file.suffix.lower() == ".json":
+                data.extend(load_json(file))
+            elif file.suffix.lower() == ".csv":
+                data.extend(load_csv(file))
+        return data
+    else:
+        raise FileNotFoundError(f"The path {path} doesn't exist")

datavane-0.1.0/src/datavane/models.py

@@ -0,0 +1,34 @@
+from dataclasses import dataclass, field
+
+
+@dataclass
+class FieldInfo:
+    """Store inferred information and statistics for a dataset field."""
+
+    name: str
+    types: set[str] = field(default_factory=set)
+    example: object | None = None
+    appearances: int = 0
+    null_count: int = 0
+    context_count: int = 0
+
+    def update(self, new_value, value_type) -> None:
+        """Update the field information with a new observed value.
+
+        The value type and appearance count are always updated. Null values
+        increment the null counter, while the first non-null value is stored
+        as the field example.
+
+        Args:
+            new_value: Value observed for the field.
+            value_type: Inferred type of the observed value.
+        """
+        self.types.add(value_type)
+        self.appearances += 1
+
+        if new_value is None:
+            self.null_count += 1
+            return
+
+        if self.example is None:
+            self.example = new_value