codepraxis 0.3.2__tar.gz → 0.4.0__tar.gz

codepraxis-0.4.0/ARCHITECTURE.md

@@ -0,0 +1,319 @@
+# How this works
+
+This explains the shape of the CLI and why it is shaped that way. It is written
+for someone new to the codebase. If you only want to *use* the tool, read the
+README or run `codepraxis guide`.
+
+---
+
+## What we are actually building
+
+CodePraxis gives candidates real engineering work instead of algorithm puzzles.
+A company sends someone a question; they open a browser and land in a real
+editor, in a real container, with real code in front of them. They fix or build
+something, and hidden tests decide whether it worked.
+
+This CLI is how a company **makes** those questions.
+
+The hard part is not the file format. The hard part is that a good question is
+hard to write, and a bad one is expensive: it either passes everyone, fails
+everyone, or measures something other than the job. So most of the design here
+is about making the good version the easy one.
+
+---
+
+## The core idea: a question is a folder
+
+```
+challenges/webhook-debug/
+  spec.md              the plan — what this tests and why
+  publish.json         catalog identity — id, title, difficulty, time limit
+  metadata.json        the workspace directory name
+  backend.conf         question type and language
+  setup.sh             installs what the question needs
+  source/              what the candidate starts from
+  ._tests/test_1.py    the tests, which the candidate never sees
+  ._course_data/
+    course_toc.json    picks the active test module
+    feature.md         the instructions tab
+solution/              the reference solution — a SIBLING, never inside
+```
+
+Two files are ours, not the runner's. `spec.md` is the plan, kept in version
+control beside the code it describes so that an edit six months later can
+recover the original intent. `publish.json` is the question's catalog identity;
+before it was adopted, those values were passed as one-off command-line flags,
+which meant a question's identity lived in someone's shell history.
+
+`solution/` sits outside the pack on purpose. That is what keeps it out of the
+archive that goes to candidates.
+
+### The rule everything rests on
+
+Every question is run **twice**:
+
+- with `source/` **plus** `solution/` overlaid — this must pass every test
+- with `source/` **alone** — this must fail
+
+The second one is what authors forget. If the starter passes, the tests do not
+discriminate and every candidate scores full marks. A question that cannot fail
+is not measuring anything.
+
+---
+
+## The lifecycle
+
+```
+plan  →  build  →  review  →  try  →  ship  →  edit
+```
+
+| Step | What happens | Where it runs |
+|---|---|---|
+| **plan** | Talk it through, agree what to test, write `spec.md`. No code. | Claude, via the plugin |
+| **build** | Write the pack, tests, setup script and reference solution. | Claude, via the plugin |
+| **review** | Audit the result and repair what fails. Runs inside build. | Claude + CLI checks |
+| **try** | Open it exactly as a candidate would. Nothing published. | CLI → platform |
+| **ship** | Publish as a draft, then promote to live. | CLI → platform |
+| **edit** | Pull a question back down, change it, publish a new version. | CLI → platform |
+
+Two of those steps are Claude workflows and four are CLI operations. That split
+matters: **judgement lives in the plugin, mechanics live in the CLI.** Deciding
+whether a question is any good is not something a Python function can do.
+Deciding whether the tests pass is not something a language model should be
+trusted with.
+
+---
+
+## Why plan mode carries most of the weight
+
+Everything after planning is mechanical. Planning is the only step where a
+wrong answer produces a bad question, so it gets two phases and two gates.
+
+**Phase one is conversation.** Claude establishes three things early, because
+everything else derives from them: how long the assessment is, what stack it
+uses, and where it is coming from — a fresh idea, an existing question in some
+other format, or a repository to mine. Then it keeps asking until it
+understands, which is three more questions for some subjects and ten for
+others.
+
+It stops asking when it can write down three things: the signal in one
+falsifiable sentence, the exact invocation and output contract, and at least
+three cases that fail for different reasons. Without a stopping rule an
+open-ended interview becomes an interrogation.
+
+**Phase two is the document.** Only written once a short sketch has been agreed,
+because producing a polished plan for the wrong question wastes everyone's time.
+
+### Gate A — can a model just solve it?
+
+Candidates have an AI agent and an LLM endpoint inside the container. So a
+question a model answers from the brief alone is not an assessment.
+
+The draft brief is handed to a model, cold, and the attempt is judged against
+the case table. If it solves it, **the plan stops** and Claude says which
+property made it trivial. Partial success is the target. A total miss usually
+means the brief is unclear rather than the question being hard.
+
+What survives the probe, strongest first: context the model has never seen
+(your repo, your schema quirk, your log format), debugging rather than
+authoring, decisions with no single right answer, integration volume, and
+hidden cases that punish the obvious approach.
+
+There is a real tension here. The clearer a brief is, the more one-shot-able it
+becomes. The resolution is to be **precise about the contract and silent about
+the approach**.
+
+### Gate B — can we actually provision it?
+
+Any stack is allowed. `setup.sh` runs on every container load, so Java, Go,
+Rust or a database server are all just installs. The gate does not grant
+permission; it checks that setup really provisions what the question needs and
+prices the cost. Only genuine impossibility stops a plan — Docker-in-Docker
+without privilege, or anything needing more than one container.
+
+---
+
+## Things about the container that change question design
+
+These are real constraints, and several of them silently break otherwise
+sensible questions.
+
+- **`curl` is not there.** It is installed during the image build and purged on
+  the way out. `wget`, pip and npm survive.
+- **Git cannot reach a network.** `git-upload-pack` and friends are removed.
+  Candidates can commit locally — and some questions grade that history — but
+  nothing clones or pushes.
+- **Setup runs on every load, by every candidate.** Not once at build time. A
+  three-minute install is three minutes of a sixty-minute assessment. It also
+  means unpinned versions rot: `pip install openai` resolves to whatever is
+  current that day, and a breaking release later fails during someone's
+  assessment rather than during ours. Pin everything.
+- **Setup runs in parallel with the first test load.** The runner imports the
+  test module right after spawning setup, which is why test modules must import
+  setup-installed packages inside methods rather than at the top.
+- **Test cases run one at a time.** A sequential loop, each case waiting for its
+  own timeout window plus the default. Total submit time is the sum of every
+  case, so a question with ten slow cases makes the candidate wait minutes on
+  every submit.
+
+Free in the image: Python 3, Node 20, gcc, g++, make, clangd, .NET 8, the ARM
+toolchain, QEMU, Renode, git, tmux, and an editor with Python and C++ tooling.
+
+---
+
+## How the code is laid out
+
+```
+src/codepraxis/
+  cli.py            argument parsing and wiring, nothing else
+  commands/         one module per user-facing action
+  domain/           what a pack is, what a result is, the runner contract
+  packio/           finding, loading, archiving packs
+  validation/       the lint rules
+  execution/
+    local/          the pure-Python harness
+    remote/         the platform client
+  reporting/        human and JSON output
+  scaffold/         templates for `codepraxis new`
+  plugin/           the Claude Code plugin, shipped inside the wheel
+```
+
+A few decisions worth knowing:
+
+**No runtime dependencies.** The local harness has to run on an author's
+machine with nothing but a Python interpreter. Even the network code uses
+`urllib` rather than `requests`. Keep it that way.
+
+**`cli.py` only wires things up.** Executors and reporters are chosen there and
+injected, so commands do not know whether they are running locally or remotely,
+or whether output is going to a terminal or to JSON.
+
+**`domain/contract.py` is a mirror.** Every constant in it has a counterpart in
+the production runner, with the source file and line recorded next to it.
+Mirrors drift, so `tests/conformance/` replays the harness against a corpus of
+real questions and asserts the verdicts. That corpus is private and lives
+outside this repository — point `PRAXIS_CONFORMANCE_PACKS` at it.
+
+---
+
+## Local and remote are a trust boundary
+
+| | Runs | Speed | Counts? |
+|---|---|---|---|
+| `validate --local` | Pure Python, on your machine | seconds | No — advisory |
+| `validate --remote` | The real runner image | ~1 min | **Yes — gates publish** |
+
+Local reproduces how the runner loads, orders and scores a question, so it
+catches most mistakes in the inner loop. It is **not** the container: it does
+not run `setup.sh`, does not have the image's packages, and has no LLM proxy.
+Anything it cannot check is reported as a note rather than passing silently.
+
+Publishing always requires a remote run. Local results never qualify, because a
+published question can be sent to candidates immediately.
+
+---
+
+## Publishing rules
+
+- **Remote validation runs first**, and the run is consumed, so one validation
+  cannot justify publishing repeatedly.
+- **A reference solution is required.** It is what proves the question is
+  solvable.
+- **Draft by default.** `--live` is deliberate.
+- **The company comes from your API key.** The CLI never sends a company id.
+  Ownership is derived server-side, so a compromised or mistyped client cannot
+  publish into someone else's catalog.
+- **`--challenge-id` adds a version** to an existing question, keeping its id,
+  assignments and history. Without it, publishing always creates a new
+  question — right the first time, a duplicate every time after.
+
+---
+
+## The command surface
+
+Everything is a subcommand. The old flag forms (`--publish`, `--list`,
+`--edit`, `--delete`, `--login`, `--install`, `--example`) still work, are
+hidden from help, and print a pointer to their replacement. They shipped in a
+released version, so CI jobs depend on them, and breaking those silently is
+worse than carrying the aliases.
+
+```
+codepraxis                  where am I, what is next
+codepraxis guide            the whole thing explained
+codepraxis login
+codepraxis install claude-plugin
+codepraxis new <name>
+codepraxis lint [question]
+codepraxis validate [question] --local | --remote
+codepraxis ship [question] [--live] [--challenge-id N]
+codepraxis list
+codepraxis edit <id> [--title ...] [--open]
+codepraxis delete <id>
+codepraxis example
+```
+
+A bare `codepraxis` prints status, not help. It used to dump the argparse
+listing — a wall of flags that tells a new author nothing about what to *do* —
+and it is the first thing everyone types, which makes it the best onboarding
+surface we have. It reads the directory and answers one question, in the spirit
+of `git status`:
+
+```
+codepraxis — authoring as Acme Corp
+
+  challenges/webhook-debug
+    plan        ready
+    pack        8 cases
+    solution    present
+    published   #214 · draft
+
+Next:
+    codepraxis ship --live challenges/webhook-debug
+```
+
+It only reports what it can determine from local files. Whether a question has
+*passed* validation is deliberately not guessed at — the CLI does not record run
+results, and inventing a state we cannot observe is worse than omitting it.
+
+Two more onboarding rules: **every command ends by printing the next one**, and
+the guide leaves out anything Claude handles for the author. The `testCases`
+contract and the pack layout are not in it, because an author using the plugin
+never writes either by hand and showing them makes the job look harder than it
+is.
+
+---
+
+## The Claude Code plugin
+
+`codepraxis install claude-plugin` writes a self-contained local marketplace
+into `.codepraxis/claude-plugin/`. Nothing is fetched from the network — the
+templates ship inside the wheel and are copied out with `importlib.resources`,
+so it works from a wheel, a zip or an editable install.
+
+```
+.codepraxis/claude-plugin/
+  .claude-plugin/marketplace.json
+  codepraxis/
+    .claude-plugin/plugin.json
+    commands/          the slash commands
+    skills/            loaded automatically when Claude touches a question
+```
+
+Installing refuses to overwrite an existing install without `--force`, because
+an author may have edited the commands and silently reverting that is worse
+than failing.
+
+The skills are split by rate of change: **pack-contract** is mechanical and
+changes when the runner changes; **question-design** is editorial and changes
+when our opinion of a good question changes. Mixing them in one file meant
+every editorial tweak touched the same file as the runner mirror.
+
+---
+
+## Where to start reading
+
+- `cli.py` — the whole surface in one file
+- `domain/contract.py` — everything the runner requires, with provenance
+- `packio/loader.py` — how a directory becomes a `Pack`
+- `execution/local/executor.py` — the harness that mirrors the runner
+- `plugin/templates/` — the prompts that drive plan and build

codepraxis-0.4.0/PKG-INFO

@@ -0,0 +1,278 @@
+Metadata-Version: 2.5
+Name: codepraxis
+Version: 0.4.0
+Summary: Author and validate CodePraxis challenge packs.
+Project-URL: Documentation, https://docs.codepraxis.com/authoring
+Author: CodePraxis
+License: Proprietary
+Keywords: assessment,authoring,challenge,codepraxis
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.9
+Provides-Extra: dev
+Requires-Dist: build>=1; extra == 'dev'
+Requires-Dist: pytest>=7; extra == 'dev'
+Requires-Dist: ruff==0.16.2; extra == 'dev'
+Requires-Dist: twine>=5; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# codepraxis
+
+Build real-world coding assessments from your own repository.
+
+```bash
+pip install codepraxis
+
+codepraxis example                    # see a real question, no account needed
+codepraxis login
+codepraxis install claude-plugin      # design and build questions with Claude
+codepraxis                            # where am I, what is next
+codepraxis guide                      # the whole thing explained
+```
+
+## What this is
+
+CodePraxis gives candidates real engineering work instead of algorithm puzzles.
+They open a browser, land in a real editor in a real container, and fix or build
+something. Hidden tests decide whether it worked.
+
+This CLI is how you make those questions — keeping them in your own git
+repository, editing them with your own tools, and checking them before they
+reach anyone.
+
+A question is a directory: starter code, a test module, instructions.
+
+## The steps
+
+```
+plan  →  build  →  try  →  ship
+```
+
+| | |
+|---|---|
+| **plan** | Talk it through, agree what to test. No code yet. |
+| **build** | Claude writes the question, tests it, and fixes what fails. |
+| **try** | Open it exactly as a candidate would. Nothing published. |
+| **ship** | Publish as a draft, then go live. |
+| **edit** | Pull one back down to change it later. |
+
+Run `codepraxis` at any point to see where you are and what to type next.
+
+## Working with Claude Code
+
+```bash
+codepraxis install claude-plugin
+```
+
+Writes a local plugin into `.codepraxis/claude-plugin/`, then prints the two
+commands that enable it. You get:
+
+- **`/codepraxis:plan`** — design a question: what it tests, what the candidate
+  starts from, the cases, how long it should take
+- **`/codepraxis:build`** — write it, run it, repair what fails
+- **`/codepraxis:validate`** — check an existing question and fix it
+- a **pack-authoring skill** that loads automatically when Claude touches a
+  question, so it already knows the contract
+
+Re-run with `--force` to overwrite an existing install.
+
+Planning refuses to proceed on a question a model can solve from the brief
+alone. Candidates have an AI agent in the container, so a question that is one
+prompt away from done measures nothing.
+
+## Doing it by hand
+
+The CLI works without Claude:
+
+```bash
+codepraxis new my-question                 # scaffold one that already passes
+codepraxis lint my-question                # static checks, no execution
+codepraxis validate my-question            # run it, fast and advisory
+codepraxis ship my-question                # validates in the runner, then publishes
+codepraxis list                            # your company's questions
+codepraxis edit 214 --open                 # update details, open a preview
+codepraxis ship my-question --challenge-id 214   # publish an edit, not a duplicate
+```
+
+## Two kinds of checking
+
+| Command | Runs | Speed | Counts? |
+|---|---|---|---|
+| `codepraxis lint` | Static rules over the files | milliseconds | Advisory |
+| `codepraxis validate --local` | Pure-Python harness on your machine | seconds | No — advisory |
+| `codepraxis validate --remote` | The real runner image, on CodePraxis | ~1 min | **Yes — gates publish** |
+
+`lint` never imports your code, so it is safe on a question you did not write
+and fast enough for every save.
+
+`--local` reproduces how the runner loads, orders and scores a question, so it
+catches most mistakes in the inner loop. It is **not** the container: it does
+not run `setup.sh` and does not have the image's package set. Anything it
+cannot check is reported as a `note` rather than silently passing. Publishing
+always requires a remote run.
+
+## The rule that matters most
+
+Every question is validated twice:
+
+- **solution** — `source/` overlaid with `solution/`. Must pass everything.
+- **starter** — `source/` alone. Must *fail*.
+
+The starter run is the one authors forget. A question whose starter already
+passes has tests that do not discriminate, and every candidate scores full
+marks.
+
+```
+webhook-debug  (local)
+  solution  18/18 passed
+  starter   0/18 passed
+  PASSED  8.9s
+```
+
+Three verdicts: **PASSED**, **FAILED**, and **INCONCLUSIVE** (this tier lacked
+the infrastructure to judge it — not a failure, and it does not fail the
+command). `--json` always emits a single document with a `packs` array.
+
+## Questions that call a model
+
+By default there is no model endpoint locally, so cases that need one are
+reported **unverifiable** rather than failed — a question is not broken just
+because your laptop has no LLM proxy. Point it at a real endpoint and they run
+for real:
+
+```bash
+export OPENAI_API_KEY=...              # or --llm-api-key
+export OPENAI_BASE_URL=...             # or --llm-base-url
+codepraxis validate my-question
+```
+
+Once a key is configured the leniency stops: a model failure is then a real
+failure, because it can be judged.
+
+## Layout
+
+```
+challenges/my-question/
+├── spec.md                  the plan — what this tests and why
+├── publish.json             title, difficulty, time limit, tech stack
+├── metadata.json            {"name": "..."} — becomes the workspace directory
+├── backend.conf             {"BACKEND": "AI", "LANGUAGE": "PYTHON"}
+├── setup.sh                 optional; installs dependencies (remote only)
+├── source/                  what the candidate starts from
+├── ._tests/test_1.py        a `testCases` class
+└── ._course_data/
+    ├── course_toc.json      selects the active test module
+    └── feature.md           the Instructions tab
+solution/                    sibling, never uploaded — the reference solution
+```
+
+`setup.sh` runs on **every** container load, not once at build time. Pin your
+versions: an unpinned install resolves to whatever is current that day, and a
+breaking release later fails during a candidate's assessment.
+
+## Authentication
+
+```bash
+codepraxis login
+```
+
+Prompts for an API key (hidden input), verifies it, and stores it at
+`~/.config/codepraxis/config.json` with `0600` permissions. It prints which
+company the key publishes as — worth reading, because that is what every
+publish is scoped to.
+
+In CI, skip the prompt:
+
+```bash
+export CODEPRAXIS_TOKEN=...
+export CODEPRAXIS_API_URL=...        # optional; defaults to the production API
+```
+
+## Publishing
+
+```bash
+codepraxis ship my-question            # draft, with confirmation
+codepraxis ship my-question --live     # straight to candidates
+codepraxis ship my-question --yes      # non-interactive, for CI
+```
+
+Publishing is deliberately strict, because a published question can be assigned
+to candidates immediately:
+
+- **Remote validation runs first.** Local results never qualify. Reuse an
+  earlier passing run with `--validation-run-id`.
+- **A reference solution is required.** It is what proves the question is
+  solvable.
+- **It publishes as a draft** unless you pass `--live`.
+- **The company comes from your API key.** The CLI never sends a company id —
+  ownership is derived server-side, so a compromised or mistyped client cannot
+  publish into someone else's catalog.
+
+### Publishing an edit
+
+Changing the *content* of a question means re-publishing it. Pass the id, or
+you get a second copy:
+
+```bash
+codepraxis ship my-question --challenge-id 214
+```
+
+With `--challenge-id` the platform adds a new version and keeps the question's
+id, assignments and history.
+
+### Deleting
+
+```bash
+codepraxis delete 214
+```
+
+Asks first, and the platform refuses once the question has been assigned to
+anyone — deleting it then would orphan attempts and reports. To take an
+assigned question out of circulation, set it back to draft with
+`codepraxis edit 214 --status draft`.
+
+## Older command forms
+
+`--publish`, `--list`, `--edit`, `--delete`, `--login`, `--install` and
+`--example` still work. They are hidden from help and print a pointer to the
+subcommand that replaced them.
+
+## Development
+
+```bash
+python3.11 -m pip install -e '.[dev]'
+pytest
+ruff check src tests scripts
+```
+
+The package has **no runtime dependencies**. The harness must run on an
+author's machine with nothing but a Python interpreter, so keep it that way —
+the remote tier uses `urllib` from the standard library for the same reason.
+
+See [ARCHITECTURE.md](ARCHITECTURE.md) for how the pieces fit together.
+
+### Conformance tests
+
+The harness mirrors the production runner's behaviour (`setupCodeBase.py`,
+`koro/test_loader.py`, `koro/test_runner.py`). Mirrors drift, so
+`tests/conformance/` replays the harness across a corpus of real questions and
+asserts the expected verdicts.
+
+That corpus is **private and lives outside this repository**:
+
+```bash
+PRAXIS_CONFORMANCE_PACKS=/path/to/question-bank pytest tests/conformance
+```
+
+Without the variable the conformance tests skip.
+
+> **Do not vendor questions into this repository.** This package is published
+> publicly. No challenge content, no reference solutions, no fixtures derived
+> from real questions. Scaffold templates must be written from scratch.