imperal-sdk 6.3.7__tar.gz → 6.4.1__tar.gz

{imperal_sdk-6.3.7 → imperal_sdk-6.4.1}/.github/workflows/publish.yml

@@ -41,7 +41,7 @@ jobs:
           cache: pip
 
       - name: Install package with dev extras
-        run: pip install -e ".[dev,fastapi,db]"
+        run: pip install -e ".[dev,fastapi,db,universal-index]"
 
       - name: Run pytest
         run: pytest -q --tb=short

{imperal_sdk-6.3.7 → imperal_sdk-6.4.1}/.github/workflows/test.yml

@@ -25,7 +25,7 @@ jobs:
           cache: pip
 
       - name: Install package with dev extras
-        run: pip install -e ".[dev,fastapi,db]"
+        run: pip install -e ".[dev,fastapi,db,universal-index]"
 
       - name: Run pytest
         run: pytest -q --tb=short

{imperal_sdk-6.3.7 → imperal_sdk-6.4.1}/CHANGELOG.md

@@ -2,6 +2,66 @@
 
 All notable changes to `imperal-sdk` are documented here.
 
+## 6.4.1 - 2026-11-04
+
+The first release of this line: the 6.4.0 tag failed its release gate in CI (the legacy `.doc` test assumed a
+reader where none is installed) and never reached PyPI. CI now tests with the `universal-index` extra.
+
+
+### Added
+- `imperal_sdk.universal`: every file kind a person works with, read into the code index (universal index phase 2).
+  - `kinds.kind_of(path)` -> `code | config | document | data | mail | image`, and `never_read(path)`: secrets by
+    name, keychains, `.gnupg`, browser profiles, VPN profiles, wallets, cloud/tool credential files -- never opened.
+  - `view.view_of(path, raw)` -> `View(text, sections, links, meta, skipped)`: Markdown, reStructuredText, HTML,
+    RTF, CSV/TSV, JSON, YAML, TOML, XML; DOCX, XLSX, PPTX, ODT, ODS, ODP; PDF (pages + bookmarks); legacy DOC/XLS/PPT
+    (a system converter when present, else OLE text runs); EML / MBOX; image metadata (Exif). A file that cannot be
+    read says why; untrusted XML never expands entities; Office files stay inside a document-wide budget.
+  - `index.read_view(root, path)` (caps by kind, iCloud placeholders never read) and `parse_view(path, view)`.
+- Code in 20+ languages (Kotlin, Java, Swift, Go, Rust, C/C++, C#, PHP, Ruby, Dart, Scala, Lua, Bash, ...) through
+  `tree-sitter-language-pack`. New extra `universal-index` (`tree-sitter-language-pack`, `pypdf`, `olefile`).
+- `FileIndex.kind`. `build_index` reads every kind; code and resolution configs keep their exact path.
+- Documents are chunked over their view's text, so search by meaning reaches what a contract or a report says.
+- Dockerfile, Makefile, Gemfile, Jenkinsfile and other build files are code.
+
+### Changed
+- **License: AGPL-3.0** (`AGPL-3.0-only`, the kernel's license) from this release on; earlier releases stay under
+  Apache-2.0. `LICENSE` holds the full GNU AGPL v3 text; every source header says AGPL-3.0.
+- **The code map is an index, never the code** (owner 2026-11-04): `sync_source` sends each file version as its facts,
+  line count, size and line hashes - never its text; a resolution config as its parsed keys (`parse_config`), never
+  the file. `EXTRACTOR_VERSION` 3: the vault asks every older version again, so the text it held is replaced.
+- The index describes what the code does without its body: `Def.doc` (a docstring or the comment above a
+  definition, first paragraph), `BlobFacts.doc` (a module docstring or the file's header comment, license headers
+  skipped) and `BlobFacts.names` (every distinct identifier and name-shaped string of the file).
+- `line_hash` / `line_hashes`: a quoted line is checked against a file's hashes.
+
+### Security
+- `build_index` never reads a never-read path, whoever passes it. The never-read list ignores case and trailing dots
+  (`.SSH`, `id_rsa.`) and checks where a link leads.
+- `kinds.never_read_at(root, path)` (and `kind_of(path, root=...)`): the check also takes the file's real absolute
+  path - a root that is itself a secret store (`~/secrets`, `~/.aws`, `~/Library/Keychains` opened as a folder) named
+  its files without the store's name and was read.
+- A FIFO, socket or device is never opened (reading a FIFO hung the build); `index.readable_file(st)`.
+- Code and configs are parsed only up to 1 MB (`RAW_CAP`): the parser holds ~100-300x the file in memory (measured: a
+  10 MB `.sql` peaked at 0.98 GB, a 1.4 MB `.py` at 0.47 GB; a 200 MB dump took 8 GB). A bigger file is skipped with
+  the reason. Chunking reads at most the cap too.
+- Windows cloud files (OneDrive and others: offline / recall-on-open / recall-on-data-access) are placeholders - never
+  read, so indexing never downloads them.
+
+### Unchanged
+- `code_index.indexable` -- what may reach the vault -- is the same: code and resolution configs only.
+
+## 6.3.8 — 2026-11-04
+
+### Fixed
+- `code_index`: one string-aware reader of resolution configs (`strip_jsonc`, `parse_jsonc`, `parse_config`).
+  Comments and trailing commas are removed outside strings only -- a regex stripper read `"@/*"` in a Next.js
+  `tsconfig.json` as a comment opener, lost `paths`, and every `@/` import fell out of the graph. The Tenant Vault
+  carries a literal copy (a test compares the source).
+- `sftp_list_tree` / `sftp_read_many` keep up to `SFTP_PARALLEL` (32) requests in flight on one connection: listing
+  walks level by level, reads go in slices. Live (Mac -> a US server, 1 572 files): listing 47 s -> 6.7 s, 200
+  files read 206 s -> 10.1 s. Results are the same (paths sorted; .gitignore rules still reach a directory's
+  children before they are filtered).
+
 ## 6.3.7 — 2026-10-21
 
 ### Changed