matterify 0.1.0__tar.gz

matterify-0.1.0/.gitignore

@@ -0,0 +1,211 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+# uv.lock is tracked for this application to ensure reproducible builds
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+#poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+#pdm.lock
+#pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+#pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Abstra
+# Abstra is an AI-powered process automation framework.
+# Ignore directories containing user credentials, local state, and settings.
+# Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#  that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#  and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#  you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Cursor
+#  Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to
+#  exclude from AI features like autocomplete and code analysis. Recommended for sensitive data
+#  refer to https://docs.cursor.com/context/ignore-files
+.cursorignore
+.cursorindexingignore
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+session-ses_2f0a.md
+
+# Benchmarks
+.benchmarks/

matterify-0.1.0/AGENTS.md

@@ -0,0 +1,202 @@
+# AGENTS.md
+
+## Project description
+`matterify` is a Python utility that recursively scans directory structures for Markdown files, extracts their embedded YAML frontmatter metadata, and aggregates all information into a structured, machine-readable JSON file for further processing.
+
+## Project Structure
+```text
+matterify/
+├── .python-version, pyproject.toml, uv.lock  # Env & Dependency management
+├── AGENTS.md, README.md, LICENSE             # Docs & Guidelines
+├── src/matterify/                            # Source (src-layout)
+│   ├── __init__.py                           # Metadata + public API entry points
+│   ├── extractor.py                          # Frontmatter extraction & aggregation logic
+│   ├── models.py                             # Frozen dataclasses (FrontmatterEntry, ScanMetadata, AggregatedResult)
+│   ├── scanner.py                           # Directory traversal with blacklist filtering
+│   ├── cli.py                                # Click CLI entry point
+│   ├── logging.py                            # Debug & console config
+│   └── utils/                                # Utility modules
+├── tests/                                    # Pytest suite
+│   ├── conftest.py                           # Fixtures
+│   ├── test_extractor.py, test_utils.py      # Unit tests
+│   └── test_cli.py                           # CLI tests
+└── docs/                                     # MkDocs source
+```
+
+# Development Workflows
+
+### UV Environment & Dependencies
+- **Sync:** `uv sync` (add `--all-extras` for dev/docs).
+- **Update:** `uv lock --upgrade`.
+- **Management:** `uv add <pkg>` (use `--dev` for dev); `uv remove <pkg>`; `uv pip list`.
+- **Strategy:** Use min constraints (e.g., `click>=8.1.0`) in `pyproject.toml`; rely on `uv.lock` for reproducibility. Avoid manual lock edits.
+
+### Execution & Lifecycle
+- **Run:** `uv run [matterify|python script.py|tool] [args]`.
+- **Project:** `uv init` (setup); `uv check` (compat-check).
+- **Dist:** `uv build` (wheel/sdist); `uv publish` (upload).
+
+### Standards & Git
+- **Versioning:** Strict SemVer (`MAJOR.MINOR.PATCH`).
+- **Commits:** Follow Conventional Commits (e.g., `feat:`, `fix:`, `chore:`).
+- **Automation:** **Never** commit autonomously; only execute on explicit user request.
+
+## Testing & QA
+
+### Quality Checks
+**Tools:** `ruff` (lint/fmt), `mypy` (types). Prefix cmds with `uv run`.
+- **Fmt/Lint:** `ruff format [--check] src/ tests/`, `ruff check src/ tests/`
+- **Types:** `mypy src/`
+- **Pre-Commit Gate:** `uv run ruff format src/ tests/ && uv run ruff check src/ tests/ && uv run mypy src/ && uv run pytest`
+
+### Tests (`uv run pytest`)
+- **Exec:** `.` (all), `-v` (verbose), `tests/[file].py[::func]` (targeted), `--cov=matterify --cov-report=html` (coverage).
+- **Structure:** `tests/` dir 1:1 mapping (`extractor.py`->`test_extractor.py`, `utils/__init__.py`->`test_utils.py`, `cli.py`->`test_cli.py`).
+- **FS Rules:** Prioritize critical paths. Use `tmp_path`. Name staging dirs `project/` (avoids `src/src/` nesting).
+- **Paths:** Stored paths include top-level prefix (`project/src/main.py`). Assert via `endswith()` or `rglob()`.
+- **Public API only:** Never import or call private symbols (names starting with `_`) from `src/` in tests. Test behaviour exclusively through the public API.
+- **No inline imports:** All imports must be at the top of the test file. `import` statements inside test functions are forbidden.
+
+## Tech Stack & Standards
+- **Runtime:** Python 3.12.3
+- **Concurrency:** `asyncio` (core)
+- **Package Mgmt:** `uv` via `pyproject.toml` (Build: `hatchling`)
+- **CLI/UI:** `click` (commands); `rich` (UI/verbose)
+- **Logging:** `structlog` (debug/structured)
+- **Parsing:** `pyyaml` (YAML Frontmatter)
+- **Quality:** `ruff` (lint/fmt); `mypy` (strict)
+- **Testing:** `pytest` (plugins: `benchmark`, `asyncio`)
+- **Docs:** `mkdocs` with Material theme
+
+## Coding Standards
+- **Typing:** Strict `mypy` for `src/`; relaxed for `tests/`.
+- **Type Aliases:** Use PEP 695 `type X = ...` (Python 3.12+). **Avoid** `TypeAlias` (ruff `UP040`).
+- **Format:** PEP8 via `ruff`; 100 char limit.
+- **Testing:** ≥1 unit test/function; use `tmp_path` for FS.
+- **UI/Logging:** CLI silent by default. Use `structlog` for internal debug logs and `rich` for verbose user feedback. **Strictly isolate** UI output from internal loggers.
+
+### Import Rules (ruff)
+- **Order (I001):** stdlib → third-party → local. Separate with one blank line. Run `uv run ruff check --fix` or `format` to resolve.
+- **Unused Imports (F401):** Remove immediately; every import must be referenced.
+- **`TYPE_CHECKING` (TC005):** Delete empty `if TYPE_CHECKING: pass` blocks; use only if containing symbols.
+- **Async-safe I/O (ASYNC240):** Never call blocking `pathlib.Path` methods inside `async def`. Wrap with `asyncio.to_thread(path.method, ...)`.
+- **Pathlib over `os` (PTH):** Use `pathlib` equivalents (e.g., `Path.unlink()`) over `os`. Avoid `os` unless no `pathlib` alternative exists.
+- **`contextlib.suppress` (SIM105):** Replace `try: ... except Error: pass` with `with contextlib.suppress(Error):`.
+
+### mypy Rules (strict)
+- **Return types:** All functions (including `__exit__`) require explicit annotations.
+- **`__exit__` signature:** Use exact typing:
+  ```python
+  def __exit__(
+      self,
+      exc_type: type[BaseException] | None,
+      exc_val: BaseException | None,
+      exc_tb: TracebackType | None,
+  ) -> None:
+  ```
+  Import `TracebackType` from `types` (use `if TYPE_CHECKING:` if preferred).
+- **`asyncio.to_thread`:** Pass bound methods directly: `asyncio.to_thread(path.read_text, encoding="utf-8")`. Avoid lambdas to preserve return-type inference.
+
+## Python API
+
+### Public Functions
+- `scan_directory(directory: Path, n_procs: int = 4, blacklist: tuple[str, ...] | None = None, compute_hash: bool = True, compute_stats: bool = True) -> AggregatedResult`: Scan directory and aggregate frontmatter using parallel workers. Returns an `AggregatedResult` dataclass.
+
+### AggregatedResult Structure
+The `scan_directory()` function returns an `AggregatedResult` dataclass:
+```python
+{
+    "metadata": ScanMetadata(...),
+    "files": list[FrontmatterEntry]
+}
+```
+
+Where `ScanMetadata` contains:
+```python
+{
+    "source_directory": str,
+    "total_files": int,
+    "files_with_frontmatter": int,
+    "files_without_frontmatter": int,
+    "errors": int,
+    "scan_duration_seconds": float,
+    "avg_duration_per_file_ms": float,
+    "throughput_files_per_second": float,
+}
+```
+
+### Public Types
+- `FrontmatterEntry`: Dataclass representing extracted frontmatter from a single file.
+- `ScanMetadata`: Dataclass containing summary statistics about a scan.
+- `AggregatedResult`: Dataclass holding metadata and file entries.
+
+### Status Values
+- `ok`: File successfully parsed with valid YAML frontmatter.
+- `illegal`: File has issues (no frontmatter, invalid YAML, or parse errors).
+
+## CLI
+
+### Usage
+
+```bash
+matterify DIRECTORY [OPTIONS]
+```
+
+**Arguments:**
+- `directory`: Directory to scan (required).
+
+**Options:**
+- `--output`, `-o`: Write JSON to file instead of stdout (if omitted, outputs to stdout).
+- `--n-procs`: Worker process count (default: auto-detect CPU cores).
+- `--verbose`, `-v`: Show progress and summary.
+- `--exclude`, `-e`: Additional directories to exclude.
+- `--debug`: Enable debug logging.
+- `--version`: Show version information.
+
+## Logging & UI (`src/matterify/logging.py`)
+- `configure_debug_logging(enabled)`: Configures `structlog`. Use `logging.CRITICAL` (50) for no-op. **Avoid `logging.CRITICAL + 1`** (causes `KeyError`).
+- `get_console(verbose)`: Returns `rich.Console()`. Verbose writes to **stdout** for `CliRunner` capture; otherwise `quiet=True`.
+
+### structlog Rules
+- **Init**: Use `structlog.get_logger(__name__)`. **Never** `logging.getLogger()`.
+- **Context**: Use kwargs: `logger.debug("msg", k=v)`. **Never** `extra={...}` (crashes on reserved keys like `name`).
+
+### Established log fields
+- `total_files`: Total number of Markdown files discovered.
+- `with_frontmatter`: Count of files with valid frontmatter.
+- `errors`: Count of files that produced errors.
+- `duration`: Total scan duration in seconds.
+- `output`: Path to the exported JSON file.
+
+## Architecture & Mechanisms
+
+### Module Overview
+- `extractor.py`: Core extraction logic - parsing YAML frontmatter, parallel processing with `ProcessPoolExecutor`, JSON export.
+- `scanner.py`: Directory traversal with blacklist filtering using `Path.walk()`.
+- `models.py`: Frozen dataclasses for type-safe data structures.
+- `logging.py`: `structlog` configuration and `rich.Console` factory.
+- `cli.py`: Click-based CLI entry point.
+
+### Extraction Pipeline
+1. `iter_markdown_files()` discovers all `.md`/`.markdown` files, respecting blacklist.
+2. `scan_directory()` distributes files across `ProcessPoolExecutor` workers.
+3. Each worker runs `extract_frontmatter()` which:
+   - Reads file content as UTF-8
+   - Checks for `---` delimiters
+   - Parses YAML block with `yaml.safe_load`
+   - Validates frontmatter is a dictionary
+   - Serializes `datetime`/`date` objects to ISO strings
+4. Results are sorted, deduplicated relative to root, and aggregated.
+
+### File Status Classification
+- `ok`: Valid YAML frontmatter found and parsed.
+- `illegal`: No frontmatter, invalid YAML, or parse error.
+
+## Docstring Rules
+- **Format:** Google Style (`Args:`, `Returns:`, `Raises:`).
+- **Markup:** Markdown ONLY; NO reST/Sphinx directives (`:class:`, etc.).
+- **Code/Links:** Backticks (single inline, triple block). MkDocs autorefs (`[MyClass][]`).
+- **Types:** Rely on Python type hints; do not duplicate in docstrings.
+- **Style:** PEP 257 imperative mood ("Return X", not "Returns X").
+- **Length:** One-liners for simple/private. Multi-line/sections ONLY for complex/public APIs. Omit redundant `Args:`/`Returns:`.
+- **Staleness:** Always update docstrings, inline comments, and class `Supported modes:` when implementing scaffolds. Treat stale "not yet implemented" text as a bug.

matterify-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Christian Gröling
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

matterify-0.1.0/PKG-INFO

@@ -0,0 +1,216 @@
+Metadata-Version: 2.4
+Name: matterify
+Version: 0.1.0
+Summary: Extract and aggregate YAML frontmatter from Markdown files into structured JSON
+Project-URL: Homepage, https://github.com/chgroeling/matterify
+Project-URL: Repository, https://github.com/chgroeling/matterify
+Author-email: Christian Gröling <contact@christiangroeling.de>
+License-File: LICENSE
+Keywords: cli,frontmatter,json,markdown,metadata,static-site,yaml
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Documentation
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: click>=8.1.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: structlog>=24.0.0
+Description-Content-Type: text/markdown
+
+# Matterify
+
+Extract and aggregate YAML frontmatter from Markdown files.
+
+[![Python version](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
+[![PyPI](https://img.shields.io/pypi/v/matterify.svg)](https://pypi.org/project/matterify/)
+
+## Quick Start
+
+```bash
+pip install matterify
+matterify ./docs -o output.json
+```
+
+## Installation
+
+```bash
+# Using uv (recommended)
+uv add matterify
+
+# Or with pip
+pip install matterify
+```
+
+## CLI Usage
+
+```bash
+matterify DIRECTORY [OPTIONS]
+```
+
+**Options:**
+- `-o, --output PATH` - Write JSON to file instead of stdout (if omitted, outputs to stdout)
+- `--n-procs INT` - Worker process count (default: auto-detect CPU cores)
+- `-v, --verbose` - Show progress and summary
+- `-e, --exclude TEXT` - Additional directories to exclude
+- `--hash / --no-hash` - Enable/disable SHA-256 hash computation
+- `--stats / --no-stats` - Enable/disable file statistics (size, modified time, access time)
+- `--debug` - Enable debug logging
+- `--version` - Show version information and exit
+
+**Examples:**
+
+```bash
+# Output to stdout (JSON)
+matterify ./docs
+
+# Output to file
+matterify ./docs -o output.json
+
+# Verbose output
+matterify ./docs --verbose
+
+# Disable hashes and file stats
+matterify ./docs --no-hash --no-stats
+
+# Exclude additional directories
+matterify ./docs -e build -e .cache
+```
+
+## Python API
+
+### Public Functions
+
+```python
+from pathlib import Path
+from matterify import (
+    scan_directory,
+)
+```
+
+#### scan_directory
+
+Scan directory and aggregate frontmatter using parallel workers. Returns an `AggregatedResult` dataclass.
+
+```python
+from pathlib import Path
+from matterify import scan_directory
+
+result = scan_directory(Path("./docs"))
+
+# AggregatedResult contains:
+# - result.metadata: ScanMetadata with scan statistics
+# - result.files: list of file entries with extraction results
+
+# Access metadata
+print(result.metadata.total_files)
+print(result.metadata.files_with_frontmatter)
+print(result.metadata.scan_duration_seconds)
+
+# Access files
+for entry in result.files:
+    print(entry.file_path, entry.status)
+    print(entry.stats.file_size if entry.stats else None)
+```
+
+### Public Types
+
+```python
+from matterify import (
+    FileEntry,
+    ScanMetadata,
+    AggregatedResult,
+)
+
+# FileEntry: extracted frontmatter from a single file
+entry: FileEntry
+
+# ScanMetadata: summary statistics about a scan
+metadata: ScanMetadata
+
+# AggregatedResult: holds metadata and file entries
+result: AggregatedResult
+```
+
+## JSON Output Structure
+
+When using CLI (stdout or `--output`), the payload has this shape:
+
+```json
+{
+  "metadata": {
+    "source_directory": "/path/to/docs",
+    "total_files": 10,
+    "files_with_frontmatter": 8,
+    "files_without_frontmatter": 2,
+    "errors": 0,
+    "scan_duration_seconds": 0.523,
+    "avg_duration_per_file_ms": 52.3,
+    "throughput_files_per_second": 19.1
+  },
+  "files": [
+    {
+      "file_path": "getting-started.md",
+      "frontmatter": {
+        "title": "Getting Started",
+        "date": "2024-01-15",
+        "tags": ["guide", "tutorial"]
+      },
+      "status": "ok",
+      "error": null,
+      "stats": {
+        "file_size": 1234,
+        "modified_time": "2024-01-15T10:30:00",
+        "access_time": "2024-01-15T10:30:00"
+      },
+      "file_hash": "abc123..."
+    }
+  ]
+}
+```
+
+`status` is either `"ok"` or `"illegal"`.
+
+## Default Exclusions
+
+The following directories are excluded from scanning by default:
+
+- `.git`
+- `.obsidian`
+- `__pycache__`
+- `.venv`
+- `venv`
+- `node_modules`
+- `.mypy_cache`
+- `.pytest_cache`
+- `.ruff_cache`
+
+Use `-e` or `--exclude` to add custom exclusions.
+
+## Development
+
+```bash
+# Install with dev dependencies
+uv sync --all-extras
+
+# Run tests
+uv run pytest
+
+# Format and lint
+uv run ruff format src/ tests/
+uv run ruff check src/ tests/
+
+# Type check
+uv run mypy src/
+```
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details.