testsuit 0.2.0__tar.gz → 0.2.2__tar.gz

{testsuit-0.2.0/src/testsuit.egg-info → testsuit-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: testsuit
-Version: 0.2.0
+Version: 0.2.2
 Summary: This toolbox helps preparing and analysing tests data, based on Pandas mainly.
 Author: Laurent ML
 License-Expression: Apache-2.0

{testsuit-0.2.0 → testsuit-0.2.2}/src/testsuit/datatools/datapack/model.py

@@ -125,6 +125,7 @@ class ComponentConfiguration(_Model):
 
 class ComponentType(_Model):
     configurations: dict[str, ComponentConfiguration]
+    description: str = ""                    # what the type is (shown in the Editor)
     # for the test sequencer
     extends: str = ""
     commands: dict[str, Any] = Field(default_factory=dict)

{testsuit-0.2.0 → testsuit-0.2.2}/src/testsuit/sequencer/README.md

@@ -249,7 +249,7 @@ The sequencer reuses the testsuit `datapack` tool (YAML dictionaries, Jinja2 tem
 | Term | Meaning |
 |---|---|
 | **Editor mode** | the part of the web UI that edits the lab and tests repositories (`server.editor`, section 13.2). |
-| **editing session** | a scratch git clone of one repository, on one branch, where the Editor writes: a **lab session** edits the inventory and types, a **workspace session** (one of `server.workspaces`) edits setups, test definitions, sub-pipelines, dictionaries and templates. |
+| **editing session** | a scratch git clone of one repository, on one branch, where the Editor writes (or, **in place**, the folder itself): a **lab session** edits the inventory and types, a **workspace session** (one of `server.workspaces`) edits setups, test definitions, sub-pipelines, dictionaries and templates. |
 | **context** | what the Editor's menus and help are built from: the types with their commands, roles, stage types, setups, test definitions and installed plugins, read again after each save. A lab session uses its own files; a workspace session uses the running lab. |
 | **palette** | in the pipeline graph, what a step can do on each logical component: its commands, with where each comes from, plus the generic and plugin step types. |
 | **flattened view** | the whole pipeline with sub-pipelines expanded and every command resolved for a setup configuration, as the run will see it. |
@@ -925,7 +925,7 @@ tsq config                                                  # settings in use, a
 - `GET /events` (Server-Sent Events: run and component state changes, log lines);
 - `GET /runs/{id}/pipeline` (phases and steps after sub-pipeline expansion, with their `needs`: the run graph of the UI);
 - Editor mode only (section 13.2):
-  - `GET /edit` (repositories, sessions, last pushes), `POST /edit/sessions` (scratch clone of a repository), `GET|DELETE /edit/sessions/{id}`;
+  - `GET /edit` (repositories, sessions, last pushes), `POST /edit/sessions` (scratch clone of a repository, or `local: true` to edit it in place), `GET|DELETE /edit/sessions/{id}`;
   - `GET /edit/sessions/{id}/files`, `GET|PUT|DELETE /edit/sessions/{id}/files/{path}` (a `PUT` of `data` merges it into the file, keeping its comments), `POST .../rename`;
   - helpers: `GET .../context` (lab types, roles, stage types, workspace contents), `GET .../schemas` (JSON Schemas with help and the lab's choices), `POST .../validate`, `.../flatten`, `.../palette`, `.../keys`, `.../preview`, `POST /edit/reachability`;
   - `POST .../datapack` (generate), `GET .../datapack/{name}[/files/{path}]`, `POST .../datapack/{name}/submit`;
@@ -947,7 +947,13 @@ The kernel never depends on the UI below: it is a web front end on the REST API,
 
 An optional mode of the same UI, enabled with `server.editor` (off by default, since it writes files). It edits the YAML files the CLI tools read, and adds nothing the files cannot express: whatever it does can also be done with a text editor and git.
 
-**Where it works.** The Editor never edits the sequencer's own lab folder. Each editing session works in a scratch clone of the repository being edited (the lab repository, or one of `server.workspaces`), on the branch it was opened on, so that a half-done change never reaches the running lab.
+**Where it works.** By default an editing session works in a scratch clone of the repository being edited (the lab repository, or one of `server.workspaces`), on the branch it was opened on, so that a half-done change never reaches the running lab.
+
+**In place.** A session can instead edit the folder itself (`in place (no clone)` on the Editor home, `"local": true` on `POST /edit/sessions`):
+- The files are saved where the sequencer reads them. A lab change still applies only on **Reload**; a test definition is used the next time a datapack is generated.
+- The folder need not be a git work tree. When it is, the session works on the checked-out branch, and review and push commit and push only the edited folder, in the user's name: other changes of that working copy are left alone. From a detached HEAD (a lab on a tag) nothing can be pushed.
+- Rebase replays the working copy's own commits on the upstream branch; it never drops them.
+- One in-place session per repository at a time; opening it again returns the same session. Closing it keeps the files.
 
 **Lab inventory.**
 - Forms for stages, ad hoc components and their variables, with a YAML view next to them.
@@ -1038,7 +1044,7 @@ src/testsuit/sequencer/
   results.py          # results datapack, large files, fetch.txt
   transports/         # local, ssh (AsyncSSH, SFTP), process groups and pid files
   api.py              # FastAPI app; serves the web UI under /ui
-  editor.py           # Editor mode: sessions on scratch clones, file kinds, checks, helpers, generate, push, rebase
+  editor.py           # Editor mode: sessions on scratch clones or in place, file kinds, checks, helpers, generate, push, rebase
   editor_api.py       # its REST routes (server.editor)
   editor_schemas.py   # JSON Schemas of the files, with help, filled with the lab's choices
   yamledit.py         # YAML written back keeping comments, key order and layout

{testsuit-0.2.0 → testsuit-0.2.2}/src/testsuit/sequencer/editor.py

@@ -1,8 +1,13 @@
 """Editor mode (README section 13.2): editing sessions on scratch clones, checks, helpers, push.
 
 An editing session is a clone of the repository holding the lab folder, or one of
-``server.workspaces``, in ``<storage.root>/editor/<session>/repo``. The sequencer's own lab
-folder is never written: a change reaches the lab when it is pushed, then reloaded on demand.
+``server.workspaces``, in ``<storage.root>/editor/<session>/repo``: the sequencer's own copy is not
+written, and a change reaches the lab when it is pushed, then reloaded on demand.
+
+An *in-place* session (``local``) edits the folder itself instead, without a clone: the files are
+saved where the sequencer reads them, with whatever else that working copy holds. The folder need
+not be a git work tree; when it is, review and push commit and push from it. A lab change still
+applies only on ``tsq lab reload``. One in-place session per repository at a time.
 
 Everything the Editor does can be done with a text editor and git: it reads and writes the same
 YAML files, and keeps them as written (``yamledit``).
@@ -82,10 +87,13 @@ class EditSession:
     upstream: str | None
     created_at: str = field(default_factory=now)
     pushed: list[dict] = field(default_factory=list)
+    local: bool = False             # in place: edits the folder itself, no clone
+    local_root: str = ""            # in place: the git work tree root, or the folder when it is not in one
+    git: bool = True                # in place: whether local_root is a git work tree
 
     @property
     def root(self) -> Path:
-        return Path(self.dir) / "repo"
+        return Path(self.local_root) if self.local else Path(self.dir) / "repo"
 
     @property
     def folder(self) -> Path:
@@ -177,7 +185,9 @@ class Editor:
                 return repo
         raise SequencerError(f"no repository '{repo_id}' (lab, or workspace0.. from server.workspaces)")
 
-    def open(self, repo_id: str, branch: str | None = None) -> EditSession:
+    def open(self, repo_id: str, branch: str | None = None, local: bool = False) -> EditSession:
+        if local:
+            return self._open_in_place(repo_id, branch)
         repo = self._repo(repo_id)
         source = Path(repo["folder"])
         src_root = _repo_root(source)
@@ -212,6 +222,49 @@ class Editor:
         self._save(session)
         return session
 
+    def _open_in_place(self, repo_id: str, branch: str | None) -> EditSession:
+        """A session on the folder itself. The one already open on that repository is returned as is."""
+        for existing in self.sessions.values():
+            if existing.local and existing.repo == repo_id:
+                if branch and branch != existing.branch:
+                    raise SequencerError(f"an in-place session works on the checked-out branch "
+                                         f"('{existing.branch or 'detached'}'), not '{branch}'")
+                return existing
+        repo = self._repo(repo_id)
+        source = Path(repo["folder"]).resolve()
+        if not source.is_dir():
+            raise SequencerError(f"no folder '{source}'")
+        try:
+            root, git = _repo_root(source).resolve(), True
+        except SequencerError:
+            root, git = source, False
+        current = _git(root, "symbolic-ref", "--quiet", "--short", "HEAD", check=False).strip() if git else ""
+        if branch and branch != current:
+            raise SequencerError(f"an in-place session works on the checked-out branch ('{current or 'detached'}'), "
+                                 f"not '{branch}': check it out first, or open a session on a clone")
+        session_id = uuid.uuid4().hex[:8]
+        folder = self.base / session_id
+        folder.mkdir(parents=True)
+        upstream = (_git(root, "remote", "get-url", "origin", check=False).strip() or None) if git else None
+        if git:                                         # datapacks generated here leave their work folder
+            exclude = Path(_git(root, "rev-parse", "--git-path", "info/exclude").strip())
+            exclude = exclude if exclude.is_absolute() else root / exclude
+            if ".current_datapack/" not in (exclude.read_text() if exclude.is_file() else ""):
+                exclude.parent.mkdir(parents=True, exist_ok=True)
+                with open(exclude, "a") as f:
+                    f.write("\n.current_datapack/\n")
+        session = EditSession(id=session_id, repo=repo_id, kind=repo["kind"], source=str(source), dir=str(folder),
+                              sub=source.relative_to(root).as_posix() if source != root else "", branch=current,
+                              base=_git(root, "rev-parse", "HEAD", check=False).strip() if git else "",
+                              upstream=upstream, local=True, local_root=str(root), git=git)
+        self.sessions[session_id] = session
+        self._save(session)
+        return session
+
+    def _need_git(self, session: EditSession) -> None:
+        if not session.git:
+            raise SequencerError(f"'{session.source}' is not a git work tree: there is nothing to review or push")
+
     def _save(self, session: EditSession) -> None:
         (Path(session.dir) / "session.json").write_text(json.dumps(asdict(session), indent=1))
 
@@ -221,12 +274,15 @@ class Editor:
         return self.sessions[session_id]
 
     def close(self, session_id: str) -> None:
+        """Drop the session: its clone, generated datapacks and notes. An in-place session's folder is kept."""
         session = self.sessions.pop(session_id, None)
         if session is not None:
             shutil.rmtree(session.dir, ignore_errors=True)
 
     def describe(self, session: EditSession) -> dict:
-        status = _git(session.root, "status", "--porcelain", "--untracked-files=all", check=False)
+        if not session.git:
+            return {**session.describe(), "changed": None, "head": None}
+        status = _git(session.root, "status", "--porcelain", "--untracked-files=all", "--", session.sub or ".", check=False)
         return {**session.describe(), "changed": len([line for line in status.splitlines() if line.strip()]),
                 "head": _git(session.root, "rev-parse", "HEAD", check=False).strip()}
 
@@ -967,9 +1023,12 @@ class Editor:
     # ── git: diff, push, rebase ──────────────────────────────────────────
 
     def diff(self, session: EditSession) -> dict:
-        _git(session.root, "add", "-A")
-        names = _git(session.root, "diff", "--cached", "--name-status", session.base)
-        patch = _git(session.root, "diff", "--cached", session.base)
+        self._need_git(session)
+        # only the edited folder: an in-place session's working copy may hold other changes, left alone
+        where = ["--", session.sub or "."]
+        _git(session.root, "add", "-A", *where)
+        names = _git(session.root, "diff", "--cached", "--name-status", session.base, *where)
+        patch = _git(session.root, "diff", "--cached", session.base, *where)
         files = [{"status": line.split("\t")[0], "path": line.split("\t")[-1]} for line in names.splitlines() if line]
         ahead = _git(session.root, "rev-list", "--count", f"{session.base}..HEAD").strip()
         return {"files": files, "patch": patch, "branch": session.branch, "base": session.base,
@@ -993,6 +1052,9 @@ class Editor:
         """Commit the session's changes as the user, tag, and push to the session's branch. On a refused
         push nothing is applied: the commit and the tag are undone, the changes stay in the session."""
         credentials = credentials or {}
+        self._need_git(session)
+        if not session.branch:
+            raise SequencerError("the working copy is on no branch (detached HEAD): check out a branch to push from it")
         if not message.strip():
             raise SequencerError("a commit message is needed")
         if not author_name.strip() or not author_email.strip():
@@ -1005,7 +1067,7 @@ class Editor:
         if not diff["files"]:
             raise SequencerError("nothing to push: no change since the session started")
         ident = ["-c", f"user.name={author_name}", "-c", f"user.email={author_email}"]
-        _git(session.root, *ident, "commit", "--quiet", "-m", message)
+        _git(session.root, *ident, "commit", "--quiet", "-m", message, "--", session.sub or ".")
         commit = _git(session.root, "rev-parse", "HEAD").strip()
         if tag:
             _git(session.root, *ident, "tag", "-a", tag, "-m", message)
@@ -1034,15 +1096,28 @@ class Editor:
     def rebase(self, session: EditSession, credentials: dict | None = None) -> dict:
         """Fetch the upstream branch and put the session's changes on top of it."""
         credentials = credentials or {}
+        self._need_git(session)
         result = subprocess.run(["git", "-c", "credential.helper=", "fetch", "--quiet", "--tags", "origin"],
                                 cwd=session.root, capture_output=True, text=True, timeout=300,
                                 env={**os.environ, "GIT_TERMINAL_PROMPT": "0", **self._push_env(credentials)})
         if result.returncode:
             return {"ok": False, "error": self._scrub(result.stderr.strip(), credentials)}
         _git(session.root, "add", "-A")
+        before = len(_git(session.root, "stash", "list").splitlines())
         stashed = _git(session.root, "stash", "push", "--quiet", "--include-untracked", "-m", "editor", check=False)
-        has_stash = bool(_git(session.root, "stash", "list").strip())
-        _git(session.root, "reset", "--quiet", "--hard", f"origin/{session.branch}")
+        has_stash = len(_git(session.root, "stash", "list").splitlines()) > before
+        if session.local:
+            # the user's working copy: its own commits are replayed on the upstream branch, never dropped
+            moved = subprocess.run(["git", "rebase", "--quiet", f"origin/{session.branch}"], cwd=session.root,
+                                   capture_output=True, text=True)
+            if moved.returncode:
+                _git(session.root, "rebase", "--abort", check=False)
+                if has_stash:
+                    _git(session.root, "stash", "pop", "--quiet", check=False)
+                return {"ok": False, "error": "the local commits do not apply on the latest revision: "
+                        + (moved.stderr or moved.stdout).strip(), "conflicts": []}
+        else:
+            _git(session.root, "reset", "--quiet", "--hard", f"origin/{session.branch}")
         session.base = _git(session.root, "rev-parse", "HEAD").strip()
         self._save(session)
         conflicts = []

{testsuit-0.2.0 → testsuit-0.2.2}/src/testsuit/sequencer/editor_api.py

@@ -39,7 +39,7 @@ def add_editor_routes(app: FastAPI, seq) -> Editor:
 
     @app.post("/edit/sessions", status_code=201)
     async def open_session(body: dict):
-        s = await run(editor.open, str(body.get("repo") or "lab"), body.get("branch"))
+        s = await run(editor.open, str(body.get("repo") or "lab"), body.get("branch"), bool(body.get("local")))
         return editor.describe(s)
 
     @app.get("/edit/sessions/{sid}")

{testsuit-0.2.0 → testsuit-0.2.2}/src/testsuit/sequencer/transports/local.py

@@ -45,6 +45,12 @@ class LocalTransport(Transport):
                     code = None
                 process.done.set_result(Result(code, process.text(), error=f"killed by {signal}" if signal else None,
                                                signal=signal))
+            except asyncio.CancelledError:
+                # The loop is shutting down (tsqd stopping, asyncio.run ending): close the process's
+                # transport now. Left to the garbage collector, it would be closed after the loop,
+                # which fails with 'Event loop is closed'.
+                proc._transport.close()
+                raise
             except Exception as e:                       # pragma: no cover - defensive
                 if not process.done.done():
                     process.done.set_result(Result(None, process.text(), error=f"transport: {e}"))