softschema 0.1.3__tar.gz → 0.2.0__tar.gz

{softschema-0.1.3 → softschema-0.2.0}/AGENTS.md

@@ -4,16 +4,16 @@ This repo teaches and implements the soft schema pattern.
 
 Start here:
 
-- [Softschema Guide](docs/softschema-guide.md): standalone concept and adoption guide
+- [softschema Guide](docs/softschema-guide.md): standalone concept and adoption guide
   for humans and agents.
-- [Softschema Spec](docs/softschema-spec.md): exact language-neutral artifact format.
+- [softschema Spec](docs/softschema-spec.md): exact language-neutral artifact format.
 - [Movie page example](examples/movie_page/README.md): complete Python-backed example.
 
 softschema ships two interchangeable implementations with the same CLI and library
 surface: Python/Pydantic (`softschema`, `softschema-py`) and TypeScript/Zod
 (`softschema`, `softschema-ts`). They are held to exact behavioral parity: same flags,
-same canonical JSON Schema sidecar, same validation results, so authoring an artifact is
-identical regardless of which you run.
+same canonical compiled JSON Schema, same validation results, so authoring an artifact
+is identical regardless of which you run.
 
 For implementer reference (only when changing a package itself):
 
@@ -63,10 +63,6 @@ Documentation rules:
 - Include the standard documentation footer in repo docs.
   Do not add it to authored softschema example artifacts.
 
-<!-- This document follows common-doc-guidelines.md.
-See github.com/jlevy/practical-prose and review guidelines before editing.
--->
-
 <!-- BEGIN TBD INTEGRATION format=f04 surface=agents-md -->
 ## tbd
 

softschema-0.2.0/PKG-INFO

@@ -0,0 +1,354 @@
+Metadata-Version: 2.4
+Name: softschema
+Version: 0.2.0
+Summary: Soft schema conventions and validation tools for Markdown/YAML artifacts
+Project-URL: Homepage, https://github.com/jlevy/softschema
+Project-URL: Repository, https://github.com/jlevy/softschema
+Project-URL: Issues, https://github.com/jlevy/softschema/issues
+Author: Joshua Levy
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Typing :: Typed
+Requires-Python: <4.0,>=3.11
+Requires-Dist: frontmatter-format>=0.3.0
+Requires-Dist: jsonschema>=4.20
+Requires-Dist: pydantic>=2
+Requires-Dist: ruamel-yaml>=0.18
+Requires-Dist: strif>=3.1.0
+Description-Content-Type: text/markdown
+
+# softschema
+
+Soft schemas: gradual, practical validation for Markdown/YAML artifacts that mix prose
+and structured data — built for humans and coding agents.
+
+## Quick Start
+
+Try it anywhere, with nothing installed but [uv](https://docs.astral.sh/uv/) or Node.
+Print the bundled example artifact and its compiled schema, then validate — the artifact
+is fully self-describing, so no flags are needed:
+
+```bash
+uvx softschema@latest docs example-artifact > spirited-away.md
+uvx softschema@latest docs example-schema > movie-page.schema.yaml
+uvx softschema@latest validate spirited-away.md
+```
+
+(Or `npx softschema@latest ...` for the Node implementation; the two are
+interchangeable.)
+
+To set up softschema in a repository with an agent, tell the agent:
+
+> Run `uvx softschema@latest --help` (for the Python implementation) or
+> `npx softschema@latest --help` (for the Node implementation) and follow the
+> instructions to set up softschema for this repo as a skill.
+
+The help output points the agent to `skill --install`, which writes discoverable
+`SKILL.md` mirrors for Codex, Claude Code, Gemini CLI, and other coding agents.
+
+## What Are Soft Schemas?
+
+Soft schemas are a practice for adding structure gradually to artifacts that mix
+flexible document context and machine-readable values.
+
+The idea is quite simple, but I’ve found it non-obvious enough that coding agents do not
+come up with this approach themselves.
+
+However if given the information and tools in this repo, soft schemas are unreasonably
+effective. Agents become far better at designing and building complex workflows that mix
+structured and unstructured data, such as document processing, data extraction,
+scientific or financial analyses, and many similar applications.
+
+**Soft schemas** are what we call the general practice.
+**softschema** is the name of this repository’s Markdown-and-YAML spec and the matching
+`softschema` CLI that implements it.
+
+## How Do Soft Schemas Work?
+
+The practice is very simple and language neutral.
+The simplest approach is to use artifacts in a process or pipeline that are Markdown
+documents with YAML frontmatter.
+The YAML carries selected structured values.
+The Markdown body stays readable for humans and agents and offers additional context.
+There are no strict rules about duplicating content between Markdown and YAML, but you
+can gradually adjust and enforce rules.
+
+A *hard* schema imposes structure up front: define a rigid contract, then reject
+anything that doesn’t fit.
+That suits data that is already uniform, but it is a poor fit for documents a human or
+agent writes, where most of the content is prose and only a few values need to be
+machine-readable.
+
+A *soft* schema lets you add structure gradually to the artifacts that pass between
+steps of a workflow.
+
+Structure runs along a spectrum, and each value moves along it only when it earns the
+move:
+
+```text
+plain Markdown prose
+  -> plain Markdown with a few YAML frontmatter values for specific fields
+  -> plain Markdown with more YAML fields and a bit of loose validation
+  -> plain Markdown with YAML frontmatter fully validated against JSON Schema (or Pydantic/Zod)
+  -> plain Markdown with separate structured data files or database records
+```
+
+The structured values live in the YAML payload, the boundary a tool reads, while the
+prose body stays unconstrained.
+Validation rules can be easily removed or changed if greater flexibility is needed.
+
+Promote a value into YAML when a tool reads it, validate it at the boundary when
+correctness matters, and tighten enforcement over time.
+Each promotion buys efficiency for some downstream consumer, so structure and efficiency
+grow together, value by value.
+
+Start with processes and data defined in Markdown with data right in the text.
+Have coding agents try them.
+Only add structure when it pays for itself.
+
+## When and Why Are Soft Schemas Useful?
+
+You should consider using soft schemas if:
+
+1. You wish to have coding agents perform complex processing workflows involving both
+   data and documents
+
+2. Some aspects of the workflow involve structured data that is processed efficiently
+   via code and some aspects are ill-defined
+
+3. The boundary between structured and unstructured data might evolve as you scale and
+   improve the workflows
+
+A plain text document offers flexible context for humans and agents.
+Structured records are far better when code or agents need to read values consistently
+and efficiently.
+
+Balancing these needs is often difficult and the source of significant complexity.
+
+Some engineers see the goal of productionizing an agent workflow as the process of
+converting it to reusable code or structured forms, like relational database schemas.
+
+But the reality is that *prematurely structuring data* before you understand the
+structures that best serve a workflow involving code and agents has a cost.
+At the same time, *poorly defined structure* has costs in consistency and efficiency.
+
+What is actually needed is **gradual addition of structure** and **flexible addition of
+textual context** at any time.
+
+Soft schemas are simple habits and conventions to make the boundary between structured
+and unstructured data easier to adjust in either direction as a workflow’s needs for
+flexibility, consistency, and efficiency evolve (code vs LLM calls).
+Keeping prose and structured values in one artifact is more convenient and
+context-efficient. A reader (human or agent) has only one place to look, and information
+can stay as loose prose until a downstream consumer needs it in more formal schemas.
+
+## The Artifact Shape
+
+The default shape is Markdown with YAML frontmatter.
+The `softschema:` block is the self-description quartet: `contract` names the payload
+contract, `schema` points at the compiled JSON Schema (relative to the document),
+`envelope` names the payload key, and `status` sets validation strictness:
+
+```markdown
+---
+title: Spirited Away (2001)
+softschema:
+  contract: example.movies:MoviePage/v1
+  schema: movie-page.schema.yaml
+  envelope: movie
+  status: enforced
+movie:
+  title: Spirited Away
+  release_year: 2001
+  runtime_minutes: 125
+  mpaa_rating: PG
+  directors:
+    - Hayao Miyazaki
+  genres: [Animation, Adventure, Family]
+  ratings:
+    imdb:
+      score: 8.6
+      total_votes: 850000
+---
+# Spirited Away (2001)
+
+*Spirited Away* is Hayao Miyazaki's animated fantasy about ten-year-old Chihiro, who
+slips into a spirit world and takes a job in a bathhouse for the gods to free her
+parents from the witch Yubaba.
+It won the 2003 Academy Award for Best Animated Feature.
+```
+
+The YAML payload is authoritative; a consumer reads it.
+The Markdown body overlaps with it but is for human readers: the prose adds context like
+the film’s Academy Award (which no structured field carries), and the full example’s
+body mirrors some YAML fields as tables for the reader’s convenience.
+
+Only the `softschema` block and the declared envelope key (`movie:` above) are
+softschema’s concern.
+Additional top-level frontmatter keys, such as the `title:` above (or `description:`,
+`tags:`, or any other host-specific metadata), are a separate concern: softschema
+neither forbids nor interprets them, so an artifact can coexist with whatever
+frontmatter conventions a static-site generator, doc indexer, or other tool already
+expects. Because the artifact declares `envelope: movie`, validation still needs no
+flags.
+
+Every key after `contract` is optional; a minimal artifact carries `contract` alone and
+binds its schema some other way (a `--schema` flag, or a host registry in library use).
+Contract IDs follow an enforced shape, `[namespace:]Name[/version]` — for example
+`example.movies:MoviePage/v1` or `com.acme.docs:IncidentReview/1.0` — naming a payload
+contract, not a class or import path.
+
+## Validate
+
+A self-describing artifact validates with no flags; flags override the document when you
+need to point a run elsewhere:
+
+```bash
+softschema validate doc.md                              # uses the document's bindings
+softschema validate doc.md --schema candidate.schema.yaml   # try a different schema
+softschema validate doc.md --envelope incident          # designate the payload key
+```
+
+`validate` reports structural (JSON Schema) and semantic (Pydantic/Zod model) results
+separately as deterministic JSON. Semantic validation loads a model with
+`--model module:Class` (Python) or `--model path:export` (Zod) — note that `--model`
+imports and executes local code, so use it only with trusted models; a compiled schema
+via `--schema` executes nothing and is the safe path for untrusted input.
+
+## Install
+
+Two supported ways to consume softschema; pick by use:
+
+- **Pin it as a dependency** for projects, CI gates, and library use (reproducible,
+  fast, offline, and the only way to `import` it):
+
+  ```bash
+  uv add --dev softschema==0.2.0        # Python
+  npm install -D softschema@0.2.0       # Node (or: bun add -d)
+  ```
+
+- **Zero-install** for one-off checks and agent bootstrap:
+
+  ```bash
+  uvx softschema@latest --help
+  npx softschema@latest --help
+  ```
+
+The rule of thumb: if softschema runs more than once, or in CI, or you import it — pin
+it. For a quick check or an agent bootstrapping with nothing installed, use a
+zero-install runner.
+See [Installation](docs/installation.md) for details, including the supply-chain
+cool-off that makes `@latest` safe to recommend.
+
+## Use as a Library
+
+Both packages expose the same surface (idiomatic per language).
+Register contracts at startup and validate at file boundaries:
+
+```python
+from pathlib import Path
+from softschema import Contract, Contracts, validate_artifact
+
+registry = Contracts()
+registry.register(Contract(id="mycorp.docs:IncidentReview/v1", model=IncidentReview,
+                           envelope_key="incident"))
+result = validate_artifact(Path("incident.md"),
+                           contract_id="mycorp.docs:IncidentReview/v1",
+                           registry=registry)
+```
+
+```ts
+import { validateArtifact } from "softschema";
+
+const result = validateArtifact("incident.md", contract);
+```
+
+A host registry’s bindings outrank the document’s own (`softschema.schema`/`envelope`),
+so a document cannot silently redirect a host’s validation; a contract registered
+without a schema path lets self-describing documents bind themselves.
+See the [softschema Guide](docs/softschema-guide.md) for the full playbooks.
+
+## Use as an Agent Skill
+
+Both packages ship the same [`SKILL.md`](https://agentskills.io) following the open
+Agent Skills standard discovered by Claude Code, Codex, Gemini CLI, Cursor, Copilot, and
+~20 other coding agents.
+Pointing an agent at the CLI is enough to bootstrap its understanding of the soft-schema
+approach: the `--help` epilog routes it to `skill --install`, a brief, and the bundled
+docs.
+
+```bash
+# Python:
+uvx softschema@latest --help            # entry point with skill setup pointers
+uvx softschema@latest skill --install   # install repo-local skill mirrors
+uvx softschema@latest skill --brief     # compact operating brief
+uvx softschema@latest docs guide        # full mental model and adoption path
+
+# TypeScript (same commands, same bundled docs/skill):
+npx softschema@latest --help
+npx softschema@latest skill --install
+npx softschema@latest skill --brief
+npx softschema@latest docs guide
+```
+
+Self-install the skill into a project so any agent working in the repo finds it natively
+(either package writes the identical mirrors):
+
+```bash
+uvx softschema@latest skill --install
+# or: npx softschema@latest skill --install
+# writes:
+#   .agents/skills/softschema/SKILL.md   (Codex, Gemini CLI, cross-agent installers)
+#   .claude/skills/softschema/SKILL.md   (Claude Code mirror)
+```
+
+Both mirrors carry a `DO NOT EDIT` marker.
+Re-run `skill --install` to refresh after upgrading the CLI.
+
+## Two Synchronized Implementations
+
+softschema ships **two complete, fully supported implementations** with the same CLI and
+library surface:
+
+- **Python / Pydantic**: [`softschema`](docs/softschema-python-design.md) on PyPI (run
+  as `softschema` or `softschema-py`).
+- **TypeScript / Zod**: [`softschema`](docs/softschema-typescript-design.md) on npm (run
+  as `softschema` or `softschema-ts`).
+
+The two are held to **exact behavioral parity**: equivalent CLI inputs, outputs, and
+flags; equivalent library APIs; the same canonical compiled JSON Schema
+(content-identical, with an equal `schema_sha256` fingerprint); and the same
+engine-neutral validation results.
+Every behavior change lands in a shared golden-test corpus first, then in both packages,
+and CI fails if their outputs or compiled schemas drift.
+They release together under the same version number on PyPI and npm.
+
+## Further Reading
+
+- [softschema Guide](docs/softschema-guide.md): the full mental model and adoption
+  playbooks.
+- [softschema Spec](docs/softschema-spec.md): the exact artifact format and validation
+  rules.
+- [Movie Page Example](examples/movie_page/README.md): the complete example backing the
+  snippets above.
+- [Installation](docs/installation.md): pinned vs zero-install, uv and Node setup.
+
+## Development and Contributing
+
+Repo setup, common commands, CI checks, the parity process, and the release workflow
+live in [Development](docs/development.md).
+The Python and TypeScript implementations must be kept in exact sync: any behavior
+change goes through the shared golden corpus first and then lands in both packages.
+
+<!-- This document follows common-doc-guidelines.md.
+See github.com/jlevy/practical-prose and review guidelines before editing.
+-->