cryptic-md 1.0.2__tar.gz

cryptic_md-1.0.2/LICENSE.md

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2024-2026 Camille Scott
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

cryptic_md-1.0.2/PKG-INFO

@@ -0,0 +1,127 @@
+Metadata-Version: 2.4
+Name: cryptic-md
+Version: 1.0.2
+Summary: LLM tools for summarizing web content into structured Obsidian notes.
+License-Expression: MIT
+License-File: LICENSE.md
+Keywords: obsidian,openai,llm,summarization,notes,markdown
+Author: Camille Scott
+Author-email: camille.scott.w@gmail.com
+Requires-Python: >=3.12,<4
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Classifier: Topic :: Utilities
+Requires-Dist: asyncinotify (>=4.0,<5.0)
+Requires-Dist: openai (>=2.36.0,<3.0.0)
+Requires-Dist: ponderosa[rich] (>=0.6.0,<0.7.0)
+Requires-Dist: python-dotenv (>=1.2.2,<2.0.0)
+Requires-Dist: python-frontmatter (>=1.1.0,<2.0.0)
+Requires-Dist: pyyaml (>=6.0,<7.0)
+Requires-Dist: rich (>=13.9.3,<14.0.0)
+Project-URL: Homepage, https://github.com/camillescott/cryptic
+Project-URL: Issues, https://github.com/camillescott/cryptic/issues
+Project-URL: Repository, https://github.com/camillescott/cryptic
+Description-Content-Type: text/markdown
+
+# cryptic
+
+![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/camillescott/cryptic/docker-publish.yml)
+![Docker Image Version](https://img.shields.io/docker/v/camillescott/cryptic?logo=docker)
+
+LLM tools for summarizing web content into structured Obsidian notes.
+
+## Features
+
+- Structured-output summarization via OpenAI, with per-category schemas (papers, articles, events, products, discussions, media, software, references).
+- Two-pane note generation: YAML frontmatter for metadata, Markdown body for content. Section layout and frontmatter mapping are declared via Pydantic field annotations.
+- Long-running service that watches a directory for new notes, processes them concurrently, and moves results to an output directory. Unmodified copies of each input are archived to a separate directory.
+- Persistent retry bookkeeping via a `cryptic_tries` frontmatter field, capped at a configurable `max_tries`.
+- Settle delay before reading new files so sources that write incrementally are picked up only once.
+- YAML configuration for model list, default model, reasoning effort, prompt text, and service directories.
+
+## Installation
+
+```sh
+poetry install
+```
+
+Set `OPENAI_API_KEY` in the environment or in a `.env` file in the project root.
+
+## Configuration
+
+Create `~/.config/cryptic/config.yaml`:
+
+```yaml
+openai:
+  models:
+    - gpt-5.4-mini
+  default_model: gpt-5.4-mini
+  default_reasoning: medium
+
+service:
+  vaults:
+    personal:
+      input_dir: ~/Obsidian/Personal/cryptic-staging
+      output_dir: ~/Obsidian/Personal/cryptic-processed
+      originals_dir: ~/Obsidian/Personal/cryptic-originals
+  max_concurrent: 3
+  max_tries: 3
+  pickup_delay_seconds: 3.0
+```
+
+Override the config path per-invocation with `--config /path/to/config.yaml`.
+
+## Usage
+
+Process a single note in place:
+
+```sh
+cryptic process note --note path/to/note.md
+```
+
+Run the service against the configured directories:
+
+```sh
+cryptic service
+```
+
+Drain the input directory once and exit (useful for batch runs):
+
+```sh
+cryptic service --once
+```
+
+Common flags available on both commands:
+
+- `--model NAME` — pick a model from `openai.models`.
+- `--reasoning {low,medium,high,xhigh}` — set reasoning effort.
+- `--config PATH` — use an alternate config file.
+
+## Docker
+
+Pre-built images are published to Docker Hub at `camillescott/cryptic`. The included `compose.yaml` is the simplest way to run the service:
+
+```sh
+export OPENAI_API_KEY=sk-...
+docker compose up -d
+```
+
+It bind-mounts `./vaults` → `/vaults` (your Obsidian tree) and `./config` → `/config` (a directory containing `config.yaml`). Paths inside `config.yaml` must be rooted at `/vaults`, for example `/vaults/personal/cryptic-staging`.
+
+To build the image locally instead of pulling:
+
+```sh
+docker build -t cryptic .
+```
+
+inotify works across bind mounts on Linux hosts. On Docker Desktop for macOS or Windows, host filesystem events don't propagate into the container.
+
+---
+
+Portions of this project's code have been written with agentic coding tools.
+

cryptic_md-1.0.2/README.md

@@ -0,0 +1,96 @@
+# cryptic
+
+![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/camillescott/cryptic/docker-publish.yml)
+![Docker Image Version](https://img.shields.io/docker/v/camillescott/cryptic?logo=docker)
+
+LLM tools for summarizing web content into structured Obsidian notes.
+
+## Features
+
+- Structured-output summarization via OpenAI, with per-category schemas (papers, articles, events, products, discussions, media, software, references).
+- Two-pane note generation: YAML frontmatter for metadata, Markdown body for content. Section layout and frontmatter mapping are declared via Pydantic field annotations.
+- Long-running service that watches a directory for new notes, processes them concurrently, and moves results to an output directory. Unmodified copies of each input are archived to a separate directory.
+- Persistent retry bookkeeping via a `cryptic_tries` frontmatter field, capped at a configurable `max_tries`.
+- Settle delay before reading new files so sources that write incrementally are picked up only once.
+- YAML configuration for model list, default model, reasoning effort, prompt text, and service directories.
+
+## Installation
+
+```sh
+poetry install
+```
+
+Set `OPENAI_API_KEY` in the environment or in a `.env` file in the project root.
+
+## Configuration
+
+Create `~/.config/cryptic/config.yaml`:
+
+```yaml
+openai:
+  models:
+    - gpt-5.4-mini
+  default_model: gpt-5.4-mini
+  default_reasoning: medium
+
+service:
+  vaults:
+    personal:
+      input_dir: ~/Obsidian/Personal/cryptic-staging
+      output_dir: ~/Obsidian/Personal/cryptic-processed
+      originals_dir: ~/Obsidian/Personal/cryptic-originals
+  max_concurrent: 3
+  max_tries: 3
+  pickup_delay_seconds: 3.0
+```
+
+Override the config path per-invocation with `--config /path/to/config.yaml`.
+
+## Usage
+
+Process a single note in place:
+
+```sh
+cryptic process note --note path/to/note.md
+```
+
+Run the service against the configured directories:
+
+```sh
+cryptic service
+```
+
+Drain the input directory once and exit (useful for batch runs):
+
+```sh
+cryptic service --once
+```
+
+Common flags available on both commands:
+
+- `--model NAME` — pick a model from `openai.models`.
+- `--reasoning {low,medium,high,xhigh}` — set reasoning effort.
+- `--config PATH` — use an alternate config file.
+
+## Docker
+
+Pre-built images are published to Docker Hub at `camillescott/cryptic`. The included `compose.yaml` is the simplest way to run the service:
+
+```sh
+export OPENAI_API_KEY=sk-...
+docker compose up -d
+```
+
+It bind-mounts `./vaults` → `/vaults` (your Obsidian tree) and `./config` → `/config` (a directory containing `config.yaml`). Paths inside `config.yaml` must be rooted at `/vaults`, for example `/vaults/personal/cryptic-staging`.
+
+To build the image locally instead of pulling:
+
+```sh
+docker build -t cryptic .
+```
+
+inotify works across bind mounts on Linux hosts. On Docker Desktop for macOS or Windows, host filesystem events don't propagate into the container.
+
+---
+
+Portions of this project's code have been written with agentic coding tools.

cryptic_md-1.0.2/cryptic/__init__.py

@@ -0,0 +1 @@
+__version__ = '1.0.2'

cryptic_md-1.0.2/cryptic/__main__.py

@@ -0,0 +1,20 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# File   : __main__.py
+# License: MIT
+# Author : Camille Scott <camille.scott.w@gmail.com>
+# Date   : 29.10.2024
+# (c) Camille Scott, 2024
+
+
+import sys
+
+from .cmds import commands
+
+
+def main():
+    return commands.run()
+
+
+if __name__ == '__main__':
+    sys.exit(main())

cryptic_md-1.0.2/cryptic/args.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# File   : args.py
+# License: MIT
+# Author : Camille Scott <camille.scott.w@gmail.com>
+# Date   : 31.10.2024
+# (c) Camille Scott, 2024
+
+from argparse import Action
+from enum import Enum
+from pathlib import Path
+
+from ponderosa import CmdTree, ArgParser, arggroup
+
+
+class EnumAction(Action):
+    """
+    Argparse action for handling Enums
+    """
+    def __init__(self, **kwargs):
+        # Pop off the type value
+        enum = kwargs.pop("type", None)
+
+        # Ensure an Enum subclass is provided
+        if enum is None:
+            raise ValueError("type must be assigned an Enum when using EnumAction")
+        if not issubclass(enum, Enum):
+            raise TypeError("type must be an Enum when using EnumAction")
+
+        # Generate choices from the Enum
+        kwargs.setdefault("choices", tuple(e.name for e in enum))
+
+        super(EnumAction, self).__init__(**kwargs)
+
+        self._enum = enum
+
+    def __call__(self, parser, namespace, values, option_string=None):
+        # Convert value back into an Enum
+        enum = self._enum[values]
+        setattr(namespace, self.dest, enum)
+
+
+commands = CmdTree()
+
+
+@commands.root.args("Config", common=True)
+def common_args(parser: ArgParser):
+    parser.add_argument('--config', type=Path, default=None,
+                        help='Path to YAML config file.')
+    parser.add_argument('--model', type=str, default=None,
+                        help='Override the configured default model.')
+    parser.add_argument('--reasoning', type=str, default=None,
+                        choices=['low', 'medium', 'high', 'xhigh'],
+                        help='Override the configured reasoning effort.')
+

cryptic_md-1.0.2/cryptic/chat.py

@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# File   : chat.py
+# License: MIT
+# Author : Camille Scott <camille.scott.w@gmail.com>
+# Date   : 23.10.2024
+# (c) Camille Scott, 2024
+
+from typing import Type
+
+from openai import AsyncOpenAI
+from openai.types.chat import ChatCompletion
+
+from .models import BaseNoteSummary, NoteSummary
+
+
+async def summarize_page(
+    client: AsyncOpenAI,
+    content: str,
+    *,
+    model: str,
+    system_prompt: str,
+    reasoning: str,
+    schema: Type[BaseNoteSummary] = NoteSummary,
+) -> tuple[BaseNoteSummary | None, ChatCompletion]:
+    completion = await client.chat.completions.parse(
+        model=model,
+        messages=[
+            {"role": "system", "content": system_prompt},
+            {"role": "user", "content": content},
+        ],
+        response_format=schema,
+        reasoning_effort=reasoning,
+    )
+    return completion.choices[0].message.parsed, completion

cryptic_md-1.0.2/cryptic/cmds.py

@@ -0,0 +1,150 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# File   : cmds.py
+# License: MIT
+# Author : Camille Scott <camille.scott.w@gmail.com>
+# Date   : 28.10.2024
+# (c) Camille Scott, 2024
+
+from argparse import Namespace
+from pathlib import Path
+import shutil
+
+from dotenv import load_dotenv
+from openai import AsyncOpenAI
+from rich.console import Console
+
+from .args import ArgParser, arggroup, commands, common_args, EnumAction
+from .chat import summarize_page
+from .config import AppConfig
+from .models import NoteSummary, PageCategory, summary_schema_from_category
+from .note import WebNote
+from . import service as service_mod
+
+
+def _resolve_model(args: Namespace, cfg: AppConfig, console: Console) -> str | None:
+    requested = args.model or cfg.openai.default_model
+    if requested not in cfg.openai.models:
+        console.print(
+            f'[red]Model {requested!r} is not in configured models: '
+            f'{cfg.openai.models}[/red]'
+        )
+        return None
+    return requested
+
+
+def _resolve_reasoning(args: Namespace, cfg: AppConfig) -> str:
+    return args.reasoning or cfg.openai.default_reasoning
+
+
+@common_args.postprocessor()
+def resolve_config(args: Namespace):
+    console = Console(stderr=True)
+    try:
+        cfg = AppConfig.load(args.config)
+    except (FileNotFoundError, ValueError) as e:
+        console.print(f'[red]{e}[/red]')
+        raise
+
+    args.model = _resolve_model(args, cfg, console)
+    if args.model is None:
+        raise ValueError('No model specified')
+
+    args.reasoning = _resolve_reasoning(args, cfg)
+    args.cfg = cfg
+
+    load_dotenv()
+
+
+@arggroup('Category')
+def category_args(parser: ArgParser):
+    parser.add_argument('--category', '-c', type=PageCategory, action=EnumAction)
+
+
+@category_args.apply()
+@commands.register('process', 'note',
+                   help='Process a note with the LLM and rewrite it.')
+async def process_note(args: Namespace):
+    console = Console(stderr=True)
+
+
+    console.log(f'Load {args.note}...')
+    note = WebNote(args.note)
+
+    if note.cryptic_processed and not args.force:
+        console.log('[red] Note already processed and not --force, exiting.')
+        return 1
+
+    if args.category:
+        schema = summary_schema_from_category(args.category)
+        console.log(f'[yellow] Forcing {schema} as Schema')
+    else:
+        schema = NoteSummary
+
+    client = AsyncOpenAI()
+    try:
+        with console.status(f'[bold blue]Wait for OpenAI response...'):
+            summary, completion = await summarize_page(
+                client,
+                note.content,
+                model=args.model,
+                system_prompt=args.cfg.prompt.text,
+                reasoning=args.reasoning,
+                schema=schema,
+            )
+    finally:
+        await client.close()
+
+    if summary is None:
+        console.print(f'[red] Error processing note!')
+        return 1
+
+    console.log(f'Processed note using {completion.usage.total_tokens} tokens.')
+    console.print(summary)
+
+    if args.backup:
+        console.log('Backup note...')
+        shutil.copy(args.note, args.note.with_suffix('.bak'))
+
+    console.log('Update and save note...')
+    note.process_summary(summary)
+    note.save()
+
+    console.rule('Processed Note')
+    note.to_console(console)
+
+    return 0
+
+
+@process_note.args()
+def _(parser: ArgParser):
+    parser.add_argument('--note', '-i', type=Path, required=True)
+    parser.add_argument('--force', '-f', default=False, action='store_true')
+    parser.add_argument('--backup', '-b', default=False, action='store_true')
+
+
+@commands.register('service',
+                   help='Watch configured vault directories and process new notes.')
+async def service_cmd(args: Namespace):
+    console = Console(stderr=True)
+
+    svc = args.cfg.require_service()
+    if args.max_concurrent is not None:
+        svc.max_concurrent = args.max_concurrent
+
+    return await service_mod.run(
+        console=console,
+        cfg=args.cfg,
+        svc=svc,
+        model=args.model,
+        reasoning=args.reasoning,
+        once=args.once,
+    )
+
+
+@service_cmd.args()
+def _(parser: ArgParser):
+    parser.add_argument('--max-concurrent', type=int, default=None,
+                        help='Override service.max_concurrent from config.')
+    parser.add_argument('--once', default=False, action='store_true',
+                        help='Drain existing files and exit instead of watching.')

cryptic_md-1.0.2/cryptic/config.py

@@ -0,0 +1,146 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# File   : config.py
+# License: MIT
+# Author : Camille Scott <camille.scott.w@gmail.com>
+
+from __future__ import annotations
+
+from importlib.resources import files as _pkg_files
+import os
+from pathlib import Path
+from typing import Literal, Self
+
+import yaml
+from pydantic import BaseModel, Field, ValidationError, model_validator
+
+
+DEFAULT_MODELS = ['gpt-5.4-mini', 'gpt-5.4-nano-2026-03-17']
+DEFAULT_MODEL = 'gpt-5.4-mini'
+
+ReasoningLevel = Literal['low', 'medium', 'high', 'xhigh']
+REASONING_LEVELS: tuple[ReasoningLevel, ...] = ('low', 'medium', 'high', 'xhigh')
+DEFAULT_REASONING: ReasoningLevel = 'medium'
+
+
+def _xdg_default_config_path() -> Path:
+    base = os.environ.get('XDG_CONFIG_HOME')
+    root = Path(base) if base else Path.home() / '.config'
+    return root / 'cryptic' / 'config.yaml'
+
+
+def _packaged_prompt_text() -> str:
+    return (_pkg_files('cryptic') / 'prompts' / 'categorize.txt').read_text(encoding='utf-8').strip()
+
+
+class OpenAICfg(BaseModel):
+    models: list[str] = Field(default_factory=lambda: list(DEFAULT_MODELS))
+    default_model: str = DEFAULT_MODEL
+    default_reasoning: ReasoningLevel = DEFAULT_REASONING
+
+    @model_validator(mode='after')
+    def _default_in_models(self) -> Self:
+        if self.default_model not in self.models:
+            raise ValueError(
+                f'default_model {self.default_model!r} is not in openai.models {self.models}'
+            )
+        return self
+
+
+class PromptCfg(BaseModel):
+    path: Path | None = None
+    text: str | None = None
+
+    @model_validator(mode='after')
+    def _resolve(self) -> Self:
+        if self.path is not None and self.text is not None:
+            raise ValueError('prompt: set exactly one of `path` or `text`, not both')
+        if self.path is not None:
+            resolved = Path(self.path).expanduser().resolve()
+            self.text = resolved.read_text(encoding='utf-8').strip()
+        return self
+
+
+class VaultCfg(BaseModel):
+    input_dir: Path
+    output_dir: Path
+    originals_dir: Path
+    name: str | None = None
+
+    @model_validator(mode='after')
+    def _expand(self) -> Self:
+        self.input_dir = Path(self.input_dir).expanduser().resolve()
+        self.output_dir = Path(self.output_dir).expanduser().resolve()
+        self.originals_dir = Path(self.originals_dir).expanduser().resolve()
+        return self
+
+
+class ServiceCfg(BaseModel):
+    vaults: dict[str, VaultCfg]
+    max_concurrent: int = 3
+    max_tries: int = 3
+    pickup_delay_seconds: float = 3.0
+
+    @model_validator(mode='after')
+    def _check(self) -> Self:
+        if not self.vaults:
+            raise ValueError('service.vaults must define at least one vault')
+        seen: dict[Path, str] = {}
+        for name, vault in self.vaults.items():
+            vault.name = name
+            if vault.input_dir in seen:
+                raise ValueError(
+                    f'vaults {seen[vault.input_dir]!r} and {name!r} share '
+                    f'input_dir {vault.input_dir}; input_dirs must be distinct'
+                )
+            seen[vault.input_dir] = name
+        if self.max_concurrent < 1:
+            raise ValueError('service.max_concurrent must be >= 1')
+        if self.max_tries < 1:
+            raise ValueError('service.max_tries must be >= 1')
+        if self.pickup_delay_seconds < 0:
+            raise ValueError('service.pickup_delay_seconds must be >= 0')
+        return self
+
+
+class AppConfig(BaseModel):
+    openai: OpenAICfg = Field(default_factory=OpenAICfg)
+    prompt: PromptCfg = Field(default_factory=PromptCfg)
+    service: ServiceCfg | None = None
+
+    @model_validator(mode='after')
+    def _default_prompt(self) -> Self:
+        if self.prompt.text is None and self.prompt.path is None:
+            self.prompt = PromptCfg(text=_packaged_prompt_text())
+        return self
+
+    @classmethod
+    def load(cls, path: Path | None) -> AppConfig:
+        if path is not None:
+            path = Path(path).expanduser().resolve()
+            if not path.exists():
+                raise FileNotFoundError(f'config file not found: {path}')
+            return cls._from_file(path)
+
+        default = _xdg_default_config_path()
+        if default.exists():
+            return cls._from_file(default)
+
+        return cls()
+
+    @classmethod
+    def _from_file(cls, path: Path) -> AppConfig:
+        with path.open('r', encoding='utf-8') as fp:
+            raw = yaml.safe_load(fp) or {}
+        try:
+            return cls.model_validate(raw)
+        except ValidationError as e:
+            raise ValueError(f'invalid config at {path}:\n{e}') from e
+
+    def require_service(self) -> ServiceCfg:
+        if self.service is None:
+            raise ValueError(
+                'service config required: add a `service:` section to your config.yaml '
+                f'(default location: {_xdg_default_config_path()})'
+            )
+        return self.service