@functional-systems/srcx 0.1.0-dev.1 → 0.1.0-dev.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +56 -1
- package/DESIGN.md +102 -34
- package/README.md +163 -12
- package/RELEASING.md +17 -8
- package/VALIDATION.md +325 -19
- package/dist/build.d.ts +73 -1
- package/dist/build.js +122 -21
- package/dist/build.js.map +1 -1
- package/dist/chunk.d.ts +2 -1
- package/dist/chunk.js +10 -9
- package/dist/chunk.js.map +1 -1
- package/dist/cli.js +55 -18
- package/dist/cli.js.map +1 -1
- package/dist/publish.d.ts +18 -0
- package/dist/publish.js +154 -19
- package/dist/publish.js.map +1 -1
- package/dist/releases.js +21 -3
- package/dist/releases.js.map +1 -1
- package/dist/remote.js +38 -9
- package/dist/remote.js.map +1 -1
- package/dist/repository.d.ts +4 -1
- package/dist/repository.js +15 -11
- package/dist/repository.js.map +1 -1
- package/dist/search.d.ts +7 -2
- package/dist/search.js +24 -4
- package/dist/search.js.map +1 -1
- package/package.json +13 -5
- package/src/build.ts +166 -24
- package/src/chunk.ts +12 -7
- package/src/cli.ts +105 -22
- package/src/publish.ts +222 -20
- package/src/releases.ts +36 -3
- package/src/remote.ts +59 -13
- package/src/repository.ts +20 -11
- package/src/search.ts +37 -8
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,59 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
- Retain allowlisted CLI evaluation failure diagnostics without raw output or credentials.
|
|
6
|
+
- Prepare an offline, unreviewed Click/Cobra question set with pinned source evidence.
|
|
7
|
+
- Default CLI search/read evaluation across two pinned public repositories, with
|
|
8
|
+
alternative answer sets, exact source verification, and complete stdout accounting.
|
|
9
|
+
- Internal reproducible lexical chunking evaluation with pinned source labels,
|
|
10
|
+
equal path-only enrichment, isolated live indexes, and token-budget metrics.
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- Opt-in managed `text-embedding-3-large` (3072 dimensions) in separate pinned
|
|
15
|
+
Collections, with paired small/large evaluation and unchanged small identities.
|
|
16
|
+
|
|
17
|
+
- Paired identifier/natural-language/mixed-query diagnostics on eight new tasks,
|
|
18
|
+
with verified candidate-rank coverage and offline stdout-budget analysis.
|
|
19
|
+
- Bounded actual-CLI lexical/semantic/hybrid evaluation on identical managed
|
|
20
|
+
corpora, with reviewed and original labels, per-question regressions, stdout
|
|
21
|
+
token counts, command timings and durable request reservations.
|
|
22
|
+
|
|
23
|
+
- Opt-in LambdaDB managed OpenAI `text-embedding-3-small` Collections, offline
|
|
24
|
+
embedding input previews, and explicit semantic/RRF hybrid search modes.
|
|
25
|
+
- Candidate validation of generated vectors separately from exact source payloads,
|
|
26
|
+
unchanged-record reuse, and a bounded synthetic managed acceptance harness.
|
|
27
|
+
|
|
28
|
+
- Persistent one-to-one Git branch / Collection Branch mapping, branch-local
|
|
29
|
+
incremental updates, and branch-selected search, resolve, and read against the
|
|
30
|
+
last published commit Tag.
|
|
31
|
+
- Remote pending import ownership and resumable in-place branch updates, including
|
|
32
|
+
shared commit Tags across branches and non-fast-forward Git branch changes.
|
|
33
|
+
- `test:live:branches` for repeatable synthetic live acceptance with retained
|
|
34
|
+
checkpoint, retry, and resource evidence.
|
|
35
|
+
|
|
36
|
+
### Changed
|
|
37
|
+
|
|
38
|
+
- Git branch imports update their fixed `git-*` writer; SHA/tag imports retain
|
|
39
|
+
manual workspaces. Existing publications and pending manual journals remain
|
|
40
|
+
usable. See README for migration and artifact behavior.
|
|
41
|
+
- Ambiguous indexed branch/Git tag names require explicit `refs/heads/...` or
|
|
42
|
+
`refs/tags/...` selectors.
|
|
43
|
+
|
|
44
|
+
### Fixed
|
|
45
|
+
|
|
46
|
+
- Fetch omitted managed vectors from query responses through the same immutable
|
|
47
|
+
Tag, preserving hit order/scores and requiring exact non-vector payload equality.
|
|
48
|
+
|
|
49
|
+
- Reject an unprepared CLI evaluation run before creating its root/lock so a
|
|
50
|
+
subsequent offline preparation can use the same path.
|
|
51
|
+
- Pin branch control before enumerating versions so a concurrent publication
|
|
52
|
+
cannot advance the selector beyond the version list being read.
|
|
53
|
+
- Reject live checkpoint reuse across different code/harness inputs and preserve
|
|
54
|
+
the original validation revision and completion time on same-input reruns.
|
|
55
|
+
|
|
56
|
+
## [0.1.0-dev.1] - 2026-09-25
|
|
57
|
+
|
|
5
58
|
### Added
|
|
6
59
|
|
|
7
60
|
- Git commit imports, syntax-aware chunks, immutable LambdaDB versions, lexical
|
|
@@ -17,4 +70,6 @@
|
|
|
17
70
|
- Pending imports resume without the previous local build artifact.
|
|
18
71
|
- Search/read `--version` selects the corpus instead of printing the CLI version.
|
|
19
72
|
|
|
20
|
-
|
|
73
|
+
Initial npm bootstrap published to the `dev` channel from commit
|
|
74
|
+
`f94f947bf593b2f8498c07e130933b0ab444f868`. This local bootstrap has no
|
|
75
|
+
GitHub Actions provenance. No stable release has been published.
|
package/DESIGN.md
CHANGED
|
@@ -18,14 +18,13 @@ supporting each result. The project name is `srcx`.
|
|
|
18
18
|
| --- | --- |
|
|
19
19
|
| Project and display name | `srcx` |
|
|
20
20
|
| Local directory | `srcx` |
|
|
21
|
-
|
|
|
21
|
+
| GitHub repository | `lambdadb/srcx` |
|
|
22
22
|
| CLI executable | `srcx` |
|
|
23
|
-
|
|
|
23
|
+
| npm package | `@functional-systems/srcx` |
|
|
24
24
|
| Short description | Version-aware code search on LambdaDB |
|
|
25
25
|
|
|
26
|
-
The
|
|
27
|
-
|
|
28
|
-
present in the initial CLI. Keep display/configuration naming centralized and
|
|
26
|
+
The public GitHub repository and npm development channel are available. Current
|
|
27
|
+
publication evidence and stable-release status are recorded in RELEASING.md. Keep display/configuration naming centralized and
|
|
29
28
|
distribution identity in `package.json`. Database IDs and the existing
|
|
30
29
|
`code-*` Collection naming and `purpose=code-search-v1` protocol marker remain
|
|
31
30
|
independent of the product name.
|
|
@@ -40,7 +39,7 @@ Confirmed boundaries from the discussion:
|
|
|
40
39
|
- Map each Git tag to a LambdaDB Alias targeting its published commit Tag.
|
|
41
40
|
Multiple Git tags for the same commit share one commit Tag and Snapshot.
|
|
42
41
|
- Keep search results and subsequent source reads pinned to the selected version.
|
|
43
|
-
- Replace a changed file's entire chunk set initially;
|
|
42
|
+
- Replace a changed file's entire chunk set initially; retain unchanged remote records.
|
|
44
43
|
- Keep core LambdaDB work, GitHub App/webhooks, UI, MCP, and a public SaaS outside
|
|
45
44
|
the first deliverable. sbrain reference documents remain unchanged.
|
|
46
45
|
- No remote repository creation, application commit, push, publication, real
|
|
@@ -72,9 +71,10 @@ This document is not a deployed-revision audit or evidence of a passing live run
|
|
|
72
71
|
|
|
73
72
|
The initial command set below is implemented; run it as `node dist/cli.js` from
|
|
74
73
|
this checkout after `npm run build`. See README for exact options and recovery.
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
74
|
+
Git branch imports now reuse persistent `git-*` Branches; explicit commit/tag
|
|
75
|
+
imports use frozen `work-*` writers. Observed-local-tag Alias synchronization is
|
|
76
|
+
implemented; automatic Git observation and authoritative pruning remain follow-up
|
|
77
|
+
work. Immutable manifest `requestedRef` is normalized to the resolved
|
|
78
78
|
commit OID; the
|
|
79
79
|
original user spelling remains in the local build artifact. The broader contract
|
|
80
80
|
below remains the target design.
|
|
@@ -379,9 +379,10 @@ readers on multiple machines do not require local registration state.
|
|
|
379
379
|
changed/deleted file, compute obsolete IDs as previous file/chunk IDs minus
|
|
380
380
|
replacement IDs. A deleted file has an empty replacement set. Obtain previous
|
|
381
381
|
IDs from the inventory, not mutable query-by-path results.
|
|
382
|
-
6. Apply the pinned per-chunk embedding policy.
|
|
383
|
-
eligible
|
|
384
|
-
|
|
382
|
+
6. Apply the pinned per-chunk embedding policy. The managed preset materializes
|
|
383
|
+
eligible `embeddingText` inputs without calling a provider locally. LambdaDB
|
|
384
|
+
generates vectors during ordinary upsert. Skip unchanged remote records and
|
|
385
|
+
validate generated vectors in the candidate Tag before publication.
|
|
385
386
|
Bound queue memory; do not load an entire repository's content into RAM.
|
|
386
387
|
7. Materialize deterministic records and deletion lists as local build artifacts.
|
|
387
388
|
Produce size-bounded inventory parts and a root manifest template. Check all
|
|
@@ -389,8 +390,10 @@ readers on multiple machines do not require local registration state.
|
|
|
389
390
|
journal supplies the separate attempt ID when sealing the final manifest.
|
|
390
391
|
|
|
391
392
|
Dry-run executes the same inventory/chunking path but performs no database mutation
|
|
392
|
-
or embedding call.
|
|
393
|
-
|
|
393
|
+
or embedding call. Report `managed` eligible chunks and `managedTokens` (estimated
|
|
394
|
+
input tokens for the complete corpus), with `embedded=0` in the local artifact.
|
|
395
|
+
These are not billed usage or incremental upload counts. Record hashes bind the
|
|
396
|
+
submitted payload; server-generated vectors are validated separately. Report every entry,
|
|
394
397
|
exclusion reason, parse status, byte size, chunk/token counts, embedding eligibility
|
|
395
398
|
and skip reasons, and change summary. Dry-run plans may show pending embedding work;
|
|
396
399
|
published chunk records may not silently treat missing required vectors as skips.
|
|
@@ -473,11 +476,12 @@ fields are omitted when absent; records of different kinds need not share them.
|
|
|
473
476
|
| Record | Additional stored fields |
|
|
474
477
|
| --- | --- |
|
|
475
478
|
| File (`kind=file`) | `path`, `blobOid`, `contentHash`, `byteLength`, `sourceText`, `language`, `parseStatus`, `chunkCount` |
|
|
476
|
-
| Chunk (`kind=chunk`) | `fileId`, `path`, parent `contentHash`, `language`, `chunkKind`, `startByte`, `endByte`, `startLine`, `endLine`, `ordinal`, `searchText`, `embeddingStatus`; optional `symbol`, `scope`, `signature`, `embeddingSkipReason`, `embeddingInputHash`, `embedding` |
|
|
479
|
+
| Chunk (`kind=chunk`) | `fileId`, `path`, parent `contentHash`, `language`, `chunkKind`, `startByte`, `endByte`, `startLine`, `endLine`, `ordinal`, `searchText`, `embeddingStatus`; optional `symbol`, `scope`, `signature`, `embeddingSkipReason`, `embeddingInputHash`, `embeddingText`, `embedding` |
|
|
477
480
|
| Corpus root (`kind=manifest`, `role=corpus`, ID `__manifest__`) | `repoId`, `indexId`, full `commitOid`, `commitTime`, `requestedRef`, full `config`, `inventoryHash`, `recordsHash`, `counts`, `buildId`, `attemptId`, `inventoryPartIds` |
|
|
478
481
|
| Inventory part (`kind=manifest`, `role=inventory`) | `partOrdinal`, `partHash`, ordered `entries` with path, file/chunk IDs, blob/content identities, inclusion/exclusion reasons, parse states, sizes, and embedding coverage |
|
|
479
482
|
| Repository control (`kind=manifest`, `role=repository`, main ID `__repo__`) | Full source identity, repo/index IDs, immutable preset/config, initialization state |
|
|
480
|
-
|
|
|
483
|
+
| Git tag control (`kind=manifest`, `role=git-ref`, main) | Full Git tag ref, encoded Alias name, observed commit, applied version and synchronization state |
|
|
484
|
+
| Git branch control (`kind=manifest`, `role=git-branch`, main) | Full Git branch ref, encoded Branch name, validated writer baseline, canonical published Tag and pending attempt identity |
|
|
481
485
|
| Validation summary (`kind=manifest`, `role=validation`, main) | Candidate name/Snapshot, build/config/attempt identity, verifier version and checked counts/hashes; no premature claim of publication |
|
|
482
486
|
|
|
483
487
|
`startByte`/`endByte` are half-open; display lines are one-based/inclusive.
|
|
@@ -486,8 +490,13 @@ content such as function, declaration, documentation, imports, or fallback; it i
|
|
|
486
490
|
separate from the file/chunk/manifest discriminator. `counts` includes included and
|
|
487
491
|
excluded files, chunks, and embedded/skipped chunks with reason totals.
|
|
488
492
|
|
|
489
|
-
|
|
490
|
-
|
|
493
|
+
The managed preset uses `embeddingStatus=managed` to identify its generation policy,
|
|
494
|
+
not local completion. Eligible chunks carry `embeddingText=searchText` and an
|
|
495
|
+
`embeddingInputHash` over the exact input and embedding configuration; ordinary
|
|
496
|
+
upserts omit `embedding`. A completed candidate must contain a nonzero, finite
|
|
497
|
+
1536-number vector for every eligible chunk. Local artifacts contain no vectors.
|
|
498
|
+
The internal fixture-only client embedder retains `embeddingStatus=embedded`;
|
|
499
|
+
its arbitrary presets cannot be published through the CLI. `embeddingStatus=skipped` requires
|
|
491
500
|
an explicit policy reason and omits the vector and input hash. The lexical-only
|
|
492
501
|
preset records `embedding-disabled`; an embedding-enabled preset records its
|
|
493
502
|
specific exclusion reason. Pending or failed required embedding work stays in the
|
|
@@ -548,10 +557,13 @@ baseline, not a proven code-specific ranking solution. An initial enrichment
|
|
|
548
557
|
candidate preserves original identifiers while adding camelCase/snake_case terms;
|
|
549
558
|
pin and evaluate that transformation before adopting it as the preset.
|
|
550
559
|
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
|
|
560
|
+
The opt-in managed preset adds indexed text `embeddingText` and a managed
|
|
561
|
+
`embedding` vector field with provider `openai`, model `text-embedding-3-small`,
|
|
562
|
+
1536 dimensions and cosine similarity. The source field, dimensions and similarity
|
|
563
|
+
are nested under the vector field's `embedding` configuration; there are no
|
|
564
|
+
top-level dimension/similarity properties. Collection discovery verifies the
|
|
565
|
+
complete resolved schema and pinned preset. Missing `embeddingText` on skipped
|
|
566
|
+
chunks, files and manifests means no vector is generated for those records. The `none` preset omits the vector index entirely.
|
|
555
567
|
Changing presets creates a separate index; do not silently mutate an existing
|
|
556
568
|
serving index. Do not write placeholder or test vectors into a real corpus.
|
|
557
569
|
|
|
@@ -740,13 +752,36 @@ choice and the finer skip rules remain evaluation decisions, not measured result
|
|
|
740
752
|
|
|
741
753
|
### Embedding reuse
|
|
742
754
|
|
|
743
|
-
|
|
744
|
-
|
|
745
|
-
|
|
746
|
-
|
|
747
|
-
|
|
748
|
-
|
|
749
|
-
|
|
755
|
+
The managed implementation reuses unchanged records from the immutable baseline:
|
|
756
|
+
compare all submitted payload fields after separately validating and removing only
|
|
757
|
+
the server-generated vector. Do not upload those unchanged records again. Direct
|
|
758
|
+
managed vector writes are forbidden, including copies fetched from an old Tag.
|
|
759
|
+
Changed files receive new chunk IDs and may require re-embedding even if some text
|
|
760
|
+
is identical. Replaying an uncertain write may also incur embedding usage. There
|
|
761
|
+
is no srcx-managed document or query vector cache and no promised server cache.
|
|
762
|
+
The older local cache remains only for internal client-embedder fixtures.
|
|
763
|
+
|
|
764
|
+
The input includes path/symbol context and source text, not commit/line/build IDs.
|
|
765
|
+
The preset pins provider/model/dimensions/similarity and input policy, but the
|
|
766
|
+
provider model name is not an immutable model revision. Validation proves vector
|
|
767
|
+
shape and payload integrity, not deterministic numerical reproduction by OpenAI.
|
|
768
|
+
|
|
769
|
+
Search remains lexical by default. Explicit `--mode semantic` uses managed
|
|
770
|
+
`knn.queryText`; `--mode hybrid` combines the existing lexical query and that kNN
|
|
771
|
+
query using LambdaDB RRF. Both legs enforce `kind=chunk` and the same optional
|
|
772
|
+
path/language filters; kNN uses a prefilter and `k=limit`. The lexical leg keeps
|
|
773
|
+
nonembedded chunks eligible. Query/fetch/list request vectors explicitly so the
|
|
774
|
+
same validation applies to pinned result handles. When a deployment omits managed
|
|
775
|
+
vectors from list responses, fetch the affected IDs from that same immutable Tag
|
|
776
|
+
and require identical non-vector payloads before vector validation. Source reads still use the
|
|
777
|
+
original file bytes, never enriched text. No relevance gain is claimed yet.
|
|
778
|
+
|
|
779
|
+
Managed source text and semantic/hybrid query text are sent through LambdaDB to
|
|
780
|
+
OpenAI and incur embedding usage plus normal LambdaDB operations/storage. Dry-run
|
|
781
|
+
and lexical queries do not request query embeddings. Use explicit opt-in and a
|
|
782
|
+
bounded synthetic acceptance run before evaluating private or larger corpora.
|
|
783
|
+
See [managed embeddings](https://docs.lambdadb.ai/guides/collections/managed-embeddings)
|
|
784
|
+
and [hybrid query](https://docs.lambdadb.ai/guides/search/hybrid) for the API contract.
|
|
750
785
|
|
|
751
786
|
## 9. Candidate validation, publication, and `versions`
|
|
752
787
|
|
|
@@ -814,10 +849,16 @@ they do not authorize the next commit on that workspace.
|
|
|
814
849
|
attempt identity and verifier version. Use an ID derived from the full commit
|
|
815
850
|
OID and candidate Snapshot ID so attempts cannot overwrite another Snapshot's
|
|
816
851
|
evidence. Confirm its readback before publication.
|
|
817
|
-
5.
|
|
818
|
-
not from the mutable Branch. Verify both Tags have
|
|
819
|
-
matching manifest.
|
|
820
|
-
|
|
852
|
+
5. For a first publication, create the deterministic published Tag **from the
|
|
853
|
+
validated candidate Tag**, not from the mutable Branch. Verify both Tags have
|
|
854
|
+
the same Snapshot ID and matching manifest. If the commit is already published,
|
|
855
|
+
validate and reuse its canonical Tag; retain this writer's validated candidate
|
|
856
|
+
separately when its Snapshot ID differs. Publication never overwrites a commit
|
|
857
|
+
Tag with another writer's snapshot.
|
|
858
|
+
6. For a tracked Git branch, update its control record with the validated writer
|
|
859
|
+
snapshot and published commit Tag, clear pending ownership, and return the
|
|
860
|
+
published version. Branch selectors keep using the preceding published Tag
|
|
861
|
+
until this control update succeeds.
|
|
821
862
|
A delayed bookkeeping write does not erase an already published immutable Tag.
|
|
822
863
|
|
|
823
864
|
An empty Snapshot or incomplete candidate never becomes published. A stale Tag
|
|
@@ -827,6 +868,33 @@ avoid repeatedly validating known stale Snapshot IDs. A payload/build defect
|
|
|
827
868
|
requires explicit same-build recovery, not endless polling. Publication must
|
|
828
869
|
finish before the same code Branch starts a later commit.
|
|
829
870
|
|
|
871
|
+
### Persistent Git branch state
|
|
872
|
+
|
|
873
|
+
Each full `refs/heads/...` name maps to one `git-<ref-digest>` writer and one
|
|
874
|
+
`branch-<full-ref-digest>` control document on main (`role=git-branch`). The control
|
|
875
|
+
record holds the last applied immutable baseline, canonical published Tag name,
|
|
876
|
+
and pending attempt/build/commit identity. Record pending ownership before branch
|
|
877
|
+
creation or corpus writes. A missing local journal cannot authorize a second
|
|
878
|
+
import over pending remote state, even if the committed head still matches the
|
|
879
|
+
old baseline. Only the same journal may resume; automatic abandoned-attempt
|
|
880
|
+
recovery remains unimplemented.
|
|
881
|
+
|
|
882
|
+
Compare the writer head with the recorded baseline snapshot before the next
|
|
883
|
+
import, then reconcile old IDs and hashes through its immutable Tag. This works
|
|
884
|
+
without the prior local artifact and supports a branch moving backward or across
|
|
885
|
+
history. Every target commit still comes from one full committed Git tree.
|
|
886
|
+
|
|
887
|
+
An already published commit does not skip updating a different tracked branch.
|
|
888
|
+
Validate that branch's own candidate, reuse the existing canonical commit Tag,
|
|
889
|
+
and retain the candidate as the writer's exact baseline when the snapshot IDs
|
|
890
|
+
differ. Never overwrite the canonical commit publication with another writer's
|
|
891
|
+
snapshot. This also applies when a branch returns to a previously indexed commit.
|
|
892
|
+
|
|
893
|
+
Branch search/resolve/read selects the last published Tag from the control record,
|
|
894
|
+
including while the next import is pending. This promises the last indexed state,
|
|
895
|
+
not the current Git tip. A first pending import has no searchable branch version.
|
|
896
|
+
An ambiguous tracked branch/Git tag short name requires an explicit heads/tags ref.
|
|
897
|
+
|
|
830
898
|
### `versions` reads LambdaDB
|
|
831
899
|
|
|
832
900
|
1. Resolve the repository Collection through remote metadata.
|
|
@@ -971,7 +1039,7 @@ Required meaningful checks:
|
|
|
971
1039
|
- Byte coverage, Unicode/BOM/CRLF/trailing-newline/long-line spans are exact.
|
|
972
1040
|
- Parse failures and huge nodes stay bounded and visible in reports.
|
|
973
1041
|
- B contains additions/replacements/deletions while A reads remain unchanged.
|
|
974
|
-
- A controlled line-shift fixture updates positions
|
|
1042
|
+
- A controlled line-shift fixture updates positions; managed imports may re-embed changed IDs, while the internal client-cache fixture verifies input-based reuse.
|
|
975
1043
|
- Different paths remain distinct even with identical bytes; reuse follows input.
|
|
976
1044
|
- Fixed-config mismatches fail before writes; fake vectors never enter a real corpus.
|
|
977
1045
|
- Explicit obsolete-ID deletion preserves replacement IDs and removes surplus chunks.
|
package/README.md
CHANGED
|
@@ -4,13 +4,24 @@ Version-aware code search on LambdaDB. Import a committed Git tree, search its
|
|
|
4
4
|
chunks, and read exact source pinned to a validated immutable version.
|
|
5
5
|
|
|
6
6
|
This is an experimental development CLI. Local fixture, fault-injection, SDK transport,
|
|
7
|
-
and opt-in live acceptance checks are available.
|
|
8
|
-
|
|
7
|
+
and opt-in live acceptance checks are available. Development builds are available
|
|
8
|
+
on npm under `@functional-systems/srcx@dev`. See [VALIDATION.md](VALIDATION.md) for the latest observed results and
|
|
9
9
|
remaining scope, and [DESIGN.md](DESIGN.md) for the full design.
|
|
10
10
|
|
|
11
|
+
## Install a development build
|
|
12
|
+
|
|
13
|
+
Requires Node.js 22.14+ and Git. These are experimental prereleases; there is no
|
|
14
|
+
stable release yet. Select the `dev` channel explicitly:
|
|
15
|
+
|
|
16
|
+
```sh
|
|
17
|
+
npm install -g @functional-systems/srcx@dev
|
|
18
|
+
srcx --version
|
|
19
|
+
srcx --help
|
|
20
|
+
```
|
|
21
|
+
|
|
11
22
|
## Run locally
|
|
12
23
|
|
|
13
|
-
Requires Node.js 22.14+ and Git. The
|
|
24
|
+
Requires Node.js 22.14+ and Git. The default preset uses lexical search with
|
|
14
25
|
`embedding=none`; no embedding account or paid API is needed.
|
|
15
26
|
|
|
16
27
|
```sh
|
|
@@ -27,11 +38,11 @@ artifacts. It does not contact LambdaDB. The model's lexical matching is a test
|
|
|
27
38
|
convenience, not a Lucene implementation or a relevance benchmark.
|
|
28
39
|
|
|
29
40
|
All commands return JSON, except help and errors. Use `node dist/cli.js` in this
|
|
30
|
-
checkout
|
|
41
|
+
checkout, or `srcx` after installing the npm package.
|
|
31
42
|
|
|
32
43
|
For development branches, CI, and installed-package checks, see
|
|
33
44
|
[CONTRIBUTING.md](CONTRIBUTING.md). Version channels, first-publication steps, and
|
|
34
|
-
|
|
45
|
+
npm automation are documented in [RELEASING.md](RELEASING.md).
|
|
35
46
|
The project is licensed under [Apache-2.0](LICENSE).
|
|
36
47
|
|
|
37
48
|
## Preview a committed version without credentials
|
|
@@ -66,6 +77,92 @@ Chunks cover the exact source bytes, carry one-based line ranges, target 800 tok
|
|
|
66
77
|
and stay below 1,500 tokens including path/symbol context. The internal window
|
|
67
78
|
baseline is available to the test/build API, not as a public storage backend.
|
|
68
79
|
|
|
80
|
+
## Opt into managed embeddings
|
|
81
|
+
|
|
82
|
+
Supported models are `text-embedding-3-small` (1536 dimensions) and
|
|
83
|
+
`text-embedding-3-large` (3072 dimensions), both cosine. Use either name with
|
|
84
|
+
`--embedding`; each has a separate pinned preset and Collection. Existing small
|
|
85
|
+
Collections keep their identity. Changing models requires a separate import.
|
|
86
|
+
|
|
87
|
+
LambdaDB generates vectors for meaningful code, tests and prose. Imports-only
|
|
88
|
+
and structural chunks remain lexical. The CLI uses LambdaDB credentials; it does not need an OpenAI key.
|
|
89
|
+
Source text sent for embedding and semantic/hybrid queries pass through LambdaDB
|
|
90
|
+
to OpenAI and incur usage charges. See [LambdaDB managed embeddings](https://docs.lambdadb.ai/guides/collections/managed-embeddings).
|
|
91
|
+
|
|
92
|
+
```sh
|
|
93
|
+
# Offline preview: no upload or embedding request.
|
|
94
|
+
srcx import --path /path/to/repo --ref main --dry-run \
|
|
95
|
+
--embedding text-embedding-3-small --output /tmp/srcx-managed-preview
|
|
96
|
+
|
|
97
|
+
# Creates a separate Collection; existing lexical Collections stay usable.
|
|
98
|
+
srcx repo add --path /path/to/repo --embedding text-embedding-3-small
|
|
99
|
+
# Use the exact collection value returned above, especially with multiple presets.
|
|
100
|
+
srcx import --repo <collection> --ref main
|
|
101
|
+
srcx search --repo <collection> --version main --query "retry failed writes" --mode hybrid
|
|
102
|
+
srcx search --repo <collection> --version main --query "retry failed writes" --mode semantic
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
Connected imports, including `--dry-run --repo`, use the selected repository's
|
|
106
|
+
pinned preset. `--embedding` on import is only for `--dry-run --path`.
|
|
107
|
+
`search` keeps `--mode lexical` as its default; semantic and hybrid require a
|
|
108
|
+
managed Collection. Hybrid uses LambdaDB RRF with identical filters on both legs.
|
|
109
|
+
Existing result handles, lexical config hashes and publication journals stay valid.
|
|
110
|
+
|
|
111
|
+
Local artifacts contain eligible `embeddingText` and input hashes, not vectors.
|
|
112
|
+
`counts.managed` and `counts.managedTokens` estimate the complete corpus's eligible
|
|
113
|
+
chunks and input tokens; `embedded=0` does not mean the server will skip them.
|
|
114
|
+
Publication requires valid generated vectors in the immutable candidate Tag,
|
|
115
|
+
plus the same exact payload/source checks as lexical publication. Unchanged
|
|
116
|
+
records are not uploaded again. Changed file chunks and uncertain write retries
|
|
117
|
+
may be embedded again; no global or query embedding cache is promised.
|
|
118
|
+
|
|
119
|
+
A small synthetic live check is available after building:
|
|
120
|
+
|
|
121
|
+
```sh
|
|
122
|
+
npm run build
|
|
123
|
+
node --env-file=.env.local scripts/live-managed.mjs
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
It uploads two synthetic commits, bounds document inputs to 10,000 estimated
|
|
127
|
+
tokens and issues four managed search queries plus lexical checks. It retains a
|
|
128
|
+
Collection, local fixture, artifacts, result handles and `.srcx/live-managed/report.json`.
|
|
129
|
+
Completed reruns preserve the original report and make no service calls; changed
|
|
130
|
+
runtime inputs require a fresh directory. Failed runs retain journals and reject
|
|
131
|
+
automatic replay. These are correctness checks, not retrieval-quality measurements.
|
|
132
|
+
|
|
133
|
+
## Evaluate retrieval quality
|
|
134
|
+
|
|
135
|
+
The internal [lexical chunking pilot](https://github.com/lambdadb/srcx/blob/develop/eval/README.md) compares syntax-aware and
|
|
136
|
+
window chunks on a fixed public corpus and 16 pinned investigation queries.
|
|
137
|
+
`npm run eval:prepare` builds and validates the comparison offline. Live evaluation
|
|
138
|
+
is an explicit separate operation and retains isolated evaluation Collections.
|
|
139
|
+
|
|
140
|
+
The [default CLI workflow evaluation](https://github.com/lambdadb/srcx/blob/develop/eval/CLI-WORKFLOW.md)
|
|
141
|
+
adds a second public repository and verifies real `search` → `read` results with
|
|
142
|
+
the unchanged CLI preset. It uses normal repository Collections and separate local
|
|
143
|
+
state; see its documented effects before opting into the live run.
|
|
144
|
+
|
|
145
|
+
The [retrieval mode comparison](https://github.com/lambdadb/srcx/blob/develop/eval/RETRIEVAL-MODES.md) runs lexical, semantic and
|
|
146
|
+
hybrid against the same managed corpora. It retains original and reviewed labels,
|
|
147
|
+
full stdout accounting, command timings and bounded embedding request estimates.
|
|
148
|
+
`npm run eval:modes:prepare` is offline; live execution is explicit and incurs
|
|
149
|
+
managed embedding and LambdaDB usage.
|
|
150
|
+
|
|
151
|
+
The [query-style diagnostic](https://github.com/lambdadb/srcx/blob/develop/eval/QUERY-STYLES.md)
|
|
152
|
+
pairs identifiers, natural-language descriptions and mixed questions on eight new
|
|
153
|
+
tasks. It adds rank-prefix and stdout-budget measurements without changing the
|
|
154
|
+
model or ranking. [Embedding model candidates](https://github.com/lambdadb/srcx/blob/develop/eval/EMBEDDING-MODELS.md)
|
|
155
|
+
separately describe possible follow-up comparisons and integration requirements.
|
|
156
|
+
The [managed model comparison](https://github.com/lambdadb/srcx/blob/develop/eval/MODEL-COMPARISON.md)
|
|
157
|
+
compares small and large on both frozen suites, including lexical controls.
|
|
158
|
+
[Results](https://github.com/lambdadb/srcx/blob/develop/eval/MODEL-COMPARISON-RESULTS.md)
|
|
159
|
+
show mixed gains and regressions; small remains the initial managed choice.
|
|
160
|
+
|
|
161
|
+
[Transfer evaluation preparation](https://github.com/lambdadb/srcx/blob/develop/eval/TRANSFER-EVAL.md) adds source-verified,
|
|
162
|
+
unreviewed Python/Go questions; no live retrieval has been run on them.
|
|
163
|
+
New evaluation roots retain [safe command failure diagnostics](https://github.com/lambdadb/srcx/blob/develop/eval/COMMAND-DIAGNOSTICS.md)
|
|
164
|
+
for explicit recovery.
|
|
165
|
+
|
|
69
166
|
## Explicit live acceptance run
|
|
70
167
|
|
|
71
168
|
The live harness accepts `LAMBDADB_BASE_URL`, `LAMBDADB_PROJECT_NAME`, and
|
|
@@ -85,6 +182,25 @@ It saves connection settings under `.srcx/live/config.json` and keeps its state
|
|
|
85
182
|
separate under `.srcx/live/state`. Rerunning resumes the saved synthetic run and
|
|
86
183
|
refuses to reuse it against a different destination.
|
|
87
184
|
|
|
185
|
+
For persistent Git branch acceptance, run `npm run test:live:branches`. It uses a
|
|
186
|
+
separate synthetic Collection and `.srcx/live-branches/` state/report, covering
|
|
187
|
+
in-place A/B updates, last-published reads during a pending update, shared commit
|
|
188
|
+
Tags across two branches, rewinds, no-op imports, and SHA-only manual imports.
|
|
189
|
+
From an isolated worktree, load the original checkout's env file explicitly:
|
|
190
|
+
|
|
191
|
+
```sh
|
|
192
|
+
npm run build
|
|
193
|
+
node --env-file=/absolute/path/to/srcx/.env.local scripts/live-branches.mjs
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
The branch harness retains resources and resumes saved checkpoints on rerun.
|
|
197
|
+
It preserves the original source revision and completion time. Checkpoints from
|
|
198
|
+
different runtime code, harness, fixture, lockfile, or Node version are rejected
|
|
199
|
+
before connecting or overwriting evidence. To validate changed code, select a new
|
|
200
|
+
`SRCX_LIVE_BRANCHES_DIR` (for example `.srcx/live-branches-next`); retain the old
|
|
201
|
+
run and any pending journal. Its per-import deadline defaults to 300 seconds and accepts
|
|
202
|
+
`SRCX_LIVE_TIMEOUT_MS` up to 600000. Do not delete pending journals to start over.
|
|
203
|
+
|
|
88
204
|
## Connected workflow
|
|
89
205
|
|
|
90
206
|
These commands create and write LambdaDB resources. Select an approved project
|
|
@@ -104,6 +220,11 @@ node dist/cli.js repo show --repo <name>
|
|
|
104
220
|
# Import the exact artifact reviewed in the credential-free preview.
|
|
105
221
|
node dist/cli.js import --repo <name> --artifact /tmp/srcx-build-a
|
|
106
222
|
node dist/cli.js versions --repo <name>
|
|
223
|
+
|
|
224
|
+
# Keep a Git branch connected to its Collection Branch.
|
|
225
|
+
node dist/cli.js import --repo <name> --ref develop
|
|
226
|
+
node dist/cli.js search --repo <name> --version develop --query retry
|
|
227
|
+
node dist/cli.js resolve --repo <name> --ref refs/heads/develop
|
|
107
228
|
node dist/cli.js search --repo <name> --version <full-commit-A> --query retry
|
|
108
229
|
node dist/cli.js read --result <result-id> --context 20
|
|
109
230
|
node dist/cli.js read --result <result-id> --full-file
|
|
@@ -141,10 +262,28 @@ are one-based and inclusive.
|
|
|
141
262
|
## Publication and recovery
|
|
142
263
|
|
|
143
264
|
The control Branch `main` holds descriptors/version summaries. `checkpoint-empty`
|
|
144
|
-
is forked before any control write.
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
265
|
+
is forked before any control write. Importing a Git branch creates or updates its
|
|
266
|
+
stable `git-*` Collection Branch, derived from the full `refs/heads/...` name.
|
|
267
|
+
Git `main` also uses a `git-*` Branch; Collection `main` remains control-only.
|
|
268
|
+
Each later import reconciles against that branch's last validated snapshot,
|
|
269
|
+
including deletions and non-fast-forward changes. Branches sharing a commit share
|
|
270
|
+
one canonical `ver-*` Tag while retaining their own writer snapshots.
|
|
271
|
+
|
|
272
|
+
`search/read --version develop` and `resolve --ref develop` select the branch's
|
|
273
|
+
last successfully published commit Tag. During an import or after a failed update,
|
|
274
|
+
they keep serving the previous published commit; the first import is unavailable
|
|
275
|
+
until publication. This is the last indexed state, not a live Git HEAD lookup.
|
|
276
|
+
Nothing polls or fetches Git automatically: run import again to advance the branch.
|
|
277
|
+
If an indexed branch and synchronized Git tag share a short name, use
|
|
278
|
+
`refs/heads/<name>` or `refs/tags/<name>` explicitly. Known Git names take precedence
|
|
279
|
+
over commit prefixes.
|
|
280
|
+
|
|
281
|
+
A commit SHA or Git tag import does not infer a containing branch. An unpublished
|
|
282
|
+
commit uses a fresh `work-*` Branch, optionally forked from a validated previous
|
|
283
|
+
manual writer; an already published commit reuses its Tag. Manual writers remain
|
|
284
|
+
frozen. Source data is never forked from a Tag. A clean tracked branch can be
|
|
285
|
+
updated from fresh local state using its remote baseline. Local build reuse is
|
|
286
|
+
cached separately for each Git branch and for manual imports.
|
|
148
287
|
|
|
149
288
|
The importer deletes obsolete IDs, upserts changed records and inventory parts in
|
|
150
289
|
bounded sequential requests, then sends `__manifest__` **alone** after every prior
|
|
@@ -156,7 +295,10 @@ an early/stale snapshot even after the marker is observed.
|
|
|
156
295
|
|
|
157
296
|
A per-Collection local lock and durable `pending.json` journal prevent another
|
|
158
297
|
commit from overwriting a partial attempt. Only one importing host is supported.
|
|
159
|
-
|
|
298
|
+
A tracked branch also records pending import ownership in control `main` before
|
|
299
|
+
writing code. If the local journal is lost, new local state refuses to overwrite
|
|
300
|
+
that pending branch even when its committed head has not advanced yet. After a
|
|
301
|
+
timeout or unknown write result, retain the artifacts and journal:
|
|
160
302
|
|
|
161
303
|
```sh
|
|
162
304
|
node dist/cli.js import --repo <name> --artifact /path/from/pending.json \
|
|
@@ -184,8 +326,17 @@ For search and resolve, a synchronized Git tag name takes precedence over a comm
|
|
|
184
326
|
prefix, matching Git import's tag-before-OID rule. A pending tag fails explicitly
|
|
185
327
|
instead of falling back to a matching commit prefix. Use `refs/tags/<name>` to
|
|
186
328
|
select a Git tag explicitly.
|
|
187
|
-
|
|
188
|
-
follow-up work; no full history import is required.
|
|
329
|
+
Automatic Git observation, authoritative pruning, branch rename/deletion handling,
|
|
330
|
+
and garbage collection remain follow-up work; no full history import is required.
|
|
331
|
+
|
|
332
|
+
Existing commit Tags and frozen `work-*` Branches remain valid. The next import
|
|
333
|
+
using a Git branch name establishes its fixed mapping, even if that commit was
|
|
334
|
+
already published. A pending journal from the previous implementation resumes on
|
|
335
|
+
its original `work-*` writer; import the branch again after recovery to establish
|
|
336
|
+
tracking. `--artifact` uses the Git branch recorded when the artifact was built,
|
|
337
|
+
with its pinned commit; it does not reread the branch's current tip. The publication
|
|
338
|
+
JSON continues to describe the canonical commit Tag: its `writer` can be the
|
|
339
|
+
original publishing branch when multiple Git branches share that commit.
|
|
189
340
|
|
|
190
341
|
## State and implementation map
|
|
191
342
|
|
package/RELEASING.md
CHANGED
|
@@ -2,12 +2,19 @@
|
|
|
2
2
|
|
|
3
3
|
## Current state
|
|
4
4
|
|
|
5
|
-
The
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
`
|
|
9
|
-
formula has been published.
|
|
10
|
-
|
|
5
|
+
The initial `@functional-systems/srcx@0.1.0-dev.1` package was published publicly
|
|
6
|
+
from reviewed commit `f94f947bf593b2f8498c07e130933b0ab444f868`. Its exact tarball
|
|
7
|
+
passed clean-install and live LambdaDB checks. The local bootstrap has no GitHub
|
|
8
|
+
Actions provenance. npm created both `dev` and `latest` at that prerelease; select
|
|
9
|
+
`@dev` explicitly. No stable/rc release or Homebrew formula has been published.
|
|
10
|
+
|
|
11
|
+
The package-specific Trusted Publisher is configured for `lambdadb/srcx` and
|
|
12
|
+
`publish.yaml`, with direct publication allowed. `NPM_DEV_PUBLISH_ENABLED=true`.
|
|
13
|
+
The first successful [OIDC publication](https://github.com/lambdadb/srcx/actions/runs/36035063463/attempts/3)
|
|
14
|
+
produced `0.1.0-dev.3` from the same commit and moved `dev` to that version.
|
|
15
|
+
Provenance identifies this repository, workflow, commit, and run. The current
|
|
16
|
+
`latest` remains the bootstrap prerelease `0.1.0-dev.1`; it is not a stable release.
|
|
17
|
+
See [VALIDATION.md](VALIDATION.md) for registry and consumer verification.
|
|
11
18
|
|
|
12
19
|
This workflow follows [LambdaDB CLI](https://github.com/lambdadb/lambdadb-cli/blob/develop/RELEASING.md).
|
|
13
20
|
The repository keeps Prettier plus strict TypeScript checks as its existing lint
|
|
@@ -26,7 +33,8 @@ version entries in `package-lock.json` together, for example using
|
|
|
26
33
|
`npm version VERSION --no-git-tag-version`. CLI `--version` reads package metadata.
|
|
27
34
|
For explicit releases, add a dated `## [VERSION] - YYYY-MM-DD` changelog section.
|
|
28
35
|
The GitHub tag must match the version, and the prerelease flag must match its channel.
|
|
29
|
-
|
|
36
|
+
After the first stable release, `latest` identifies the stable npm release;
|
|
37
|
+
the bootstrap exception is recorded above.
|
|
30
38
|
|
|
31
39
|
The checked-in development base starts at `0.1.0-dev.1`. During an eligible push,
|
|
32
40
|
`scripts/dev-release.mjs` replaces `N` with the full history's first-parent commit
|
|
@@ -64,7 +72,8 @@ Do not republish, overwrite versions, or move release tags to work around lag.
|
|
|
64
72
|
|
|
65
73
|
## First publication and Trusted Publisher setup
|
|
66
74
|
|
|
67
|
-
|
|
75
|
+
The initial bootstrap is complete. The procedure below documents first-time setup
|
|
76
|
+
and the checks required before declaring automatic publication operational:
|
|
68
77
|
|
|
69
78
|
1. Verify npm ownership for the `functional-systems` scope and the package name.
|
|
70
79
|
GitHub organization membership does not establish npm permissions.
|