loki-mode 7.79.0 → 7.80.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/SKILL.md +2 -2
- package/VERSION +1 -1
- package/autonomy/loki +140 -2
- package/autonomy/run.sh +3 -0
- package/dashboard/__init__.py +1 -1
- package/dashboard/registry.py +212 -0
- package/dashboard/server.py +236 -0
- package/dashboard/static/index.html +383 -150
- package/docs/ENTERPRISE-IDENTITY-ROADMAP.md +206 -0
- package/docs/INSTALLATION.md +2 -2
- package/loki-ts/dist/loki.js +2 -2
- package/lokistore/__init__.py +65 -0
- package/lokistore/base.py +172 -0
- package/lokistore/cloud.py +305 -0
- package/lokistore/factory.py +187 -0
- package/lokistore/local.py +219 -0
- package/mcp/__init__.py +1 -1
- package/memory/retrieval.py +147 -0
- package/memory/tree_index.py +499 -0
- package/memory/tree_search.py +305 -0
- package/package.json +2 -1
- package/plugins/loki-mode/.claude-plugin/plugin.json +1 -1
|
@@ -0,0 +1,206 @@
|
|
|
1
|
+
# Enterprise Identity Roadmap
|
|
2
|
+
|
|
3
|
+
This document scopes Loki Mode's enterprise identity surface honestly. It
|
|
4
|
+
separates what is shipped today from what is roadmap. Nothing in the
|
|
5
|
+
"Roadmap" section ships today. Where a capability is not built, it is labeled
|
|
6
|
+
as such, with a real effort estimate and the identity-provider (IdP) or test
|
|
7
|
+
infrastructure it would require.
|
|
8
|
+
|
|
9
|
+
The intent is to give a precise, non-aspirational picture so that marketing,
|
|
10
|
+
sales, and docs never overstate the enterprise identity story.
|
|
11
|
+
|
|
12
|
+
## 1. Current state (shipped, verified)
|
|
13
|
+
|
|
14
|
+
Everything in this section is backed by code in `dashboard/auth.py` and
|
|
15
|
+
`dashboard/server.py`. Function names are cited so claims can be checked.
|
|
16
|
+
|
|
17
|
+
### Token authentication
|
|
18
|
+
|
|
19
|
+
- Opt-in via `LOKI_ENTERPRISE_AUTH=true` (off by default).
|
|
20
|
+
See `ENTERPRISE_AUTH_ENABLED` and `is_enterprise_mode()` in
|
|
21
|
+
`dashboard/auth.py`.
|
|
22
|
+
- API tokens are minted, hashed (per-token random salt, SHA-256), revoked,
|
|
23
|
+
deleted, and listed:
|
|
24
|
+
`generate_token()`, `revoke_token()`, `delete_token()`, `list_tokens()`,
|
|
25
|
+
`validate_token()`. Tokens are stored at `~/.loki/dashboard/tokens.json`
|
|
26
|
+
with enforced `0600` permissions (`_save_tokens()`).
|
|
27
|
+
- Token validation iterates all entries with a constant-time compare to avoid
|
|
28
|
+
leaking token count via timing (`validate_token()` plus
|
|
29
|
+
`_constant_time_compare()`).
|
|
30
|
+
|
|
31
|
+
### OIDC bearer-token validation (the closest foundation for SSO)
|
|
32
|
+
|
|
33
|
+
- Opt-in via `LOKI_OIDC_ISSUER` + `LOKI_OIDC_CLIENT_ID`.
|
|
34
|
+
See `OIDC_ENABLED` and `is_oidc_mode()` in `dashboard/auth.py`.
|
|
35
|
+
- Inbound JWTs are validated by `validate_oidc_token()`. When PyJWT +
|
|
36
|
+
cryptography are installed, signatures are cryptographically verified
|
|
37
|
+
(RS256/RS384/RS512) against the provider's JWKS endpoint, with issuer and
|
|
38
|
+
audience checks (`_get_oidc_config()`, `_get_jwks()`).
|
|
39
|
+
- Without PyJWT, tokens are rejected unless `LOKI_OIDC_SKIP_SIGNATURE_VERIFY`
|
|
40
|
+
is explicitly set (insecure, local-testing only, loudly logged as critical).
|
|
41
|
+
- Role/group claims are mapped to Loki roles by `_scopes_from_claims()` /
|
|
42
|
+
`_collect_role_claims()`, supporting generic `roles`/`groups`, Keycloak
|
|
43
|
+
`realm_access.roles`, AWS Cognito `cognito:groups`, and a configurable
|
|
44
|
+
claim (`LOKI_OIDC_ROLES_CLAIM`). Unrecognized claims fall back to the
|
|
45
|
+
least-privileged default role (`_default_oidc_role()`, default `viewer`),
|
|
46
|
+
never admin.
|
|
47
|
+
|
|
48
|
+
IMPORTANT distinction: this is server-side validation of a bearer JWT that
|
|
49
|
+
some other system obtained. Loki does NOT implement a browser login flow,
|
|
50
|
+
an authorization-code exchange, or an end-user SSO redirect. A separate
|
|
51
|
+
component must perform the user-facing sign-in and present the resulting
|
|
52
|
+
JWT to Loki.
|
|
53
|
+
|
|
54
|
+
### Scopes and predefined roles (read/control style authorization)
|
|
55
|
+
|
|
56
|
+
- Four predefined roles in `ROLES`: `admin`, `operator`, `viewer`,
|
|
57
|
+
`auditor`. Scope hierarchy (`_SCOPE_HIERARCHY`, `has_scope()`):
|
|
58
|
+
`*` -> `control` -> `write` -> `read`, plus `audit`/`admin`.
|
|
59
|
+
- Endpoint enforcement via the `require_scope()` dependency factory and the
|
|
60
|
+
`get_current_token()` FastAPI dependency. When neither auth mode is enabled,
|
|
61
|
+
access is anonymous (local-first default).
|
|
62
|
+
|
|
63
|
+
### Dashboard transport gating (added in Release A)
|
|
64
|
+
|
|
65
|
+
- WebSocket auth gating: the dashboard validates a bearer token (header or
|
|
66
|
+
`?token=`/`?access_token=` query for browser clients) at the mount
|
|
67
|
+
boundary before delegating, mirroring the HTTP `get_current_token` order
|
|
68
|
+
(OIDC first, then loki token). See `_MountAuthGuard._validate_ws_token()`
|
|
69
|
+
and `_ws_token_from_scope()` in `dashboard/server.py`.
|
|
70
|
+
- REST scope consistency: `/api/memory` and `/api/collab` endpoints were
|
|
71
|
+
brought in line with the read/control scope model so they are not reachable
|
|
72
|
+
unauthenticated when enterprise auth is on.
|
|
73
|
+
- Webhook/trigger HMAC: the external trigger surface requires a shared secret
|
|
74
|
+
and rejects unsigned/mismatched requests (constant-time compare via
|
|
75
|
+
`hmac.compare_digest`, surfaced through `_constant_time_compare()`).
|
|
76
|
+
|
|
77
|
+
### Tenant isolation (data-plane, present but minimal)
|
|
78
|
+
|
|
79
|
+
- A `Tenant` model exists (`dashboard/models.py`: `class Tenant`, with
|
|
80
|
+
`Project.tenant_id` foreign keys) and the v2 API enforces a tenant boundary
|
|
81
|
+
derived from a trusted, server-validated `tenant:<id>` scope on the token
|
|
82
|
+
(`dashboard/api_v2.py`: `TENANT_SCOPE_PREFIX`). A non-admin token is pinned
|
|
83
|
+
to one tenant; cross-tenant requests are denied with 403; an un-scoped
|
|
84
|
+
token reaches no tenant-scoped resource.
|
|
85
|
+
- This is project-level data isolation keyed off a token scope. It is NOT a
|
|
86
|
+
full tenant RBAC system (no roles per tenant, no per-tenant policy
|
|
87
|
+
administration, no tenant lifecycle management UI). See the roadmap below.
|
|
88
|
+
|
|
89
|
+
### What the infrastructure OIDC is (and is NOT)
|
|
90
|
+
|
|
91
|
+
The OIDC referenced in the Terraform and Helm assets is workload identity
|
|
92
|
+
(IRSA on AWS / Workload Identity on GCP) used so the running pods can assume
|
|
93
|
+
cloud roles. That is machine-to-cloud authentication, not end-user SSO.
|
|
94
|
+
|
|
95
|
+
Likewise, Kubernetes RBAC and NetworkPolicy in the Helm chart govern what the
|
|
96
|
+
cluster service accounts and pods may do. They are cluster-plane controls and
|
|
97
|
+
are unrelated to product-level or per-tenant authorization for Loki users.
|
|
98
|
+
Do not conflate cluster RBAC with application RBAC.
|
|
99
|
+
|
|
100
|
+
## 2. Roadmap (NOT built)
|
|
101
|
+
|
|
102
|
+
None of the items below are implemented. Each lists honest scope, a rough
|
|
103
|
+
effort estimate, and the IdP or test infrastructure required.
|
|
104
|
+
|
|
105
|
+
### SSO / SAML (browser sign-in flow)
|
|
106
|
+
|
|
107
|
+
- Scope: a user-facing single sign-on flow. Two sub-paths:
|
|
108
|
+
- OIDC authorization-code login (closer to today's code): a browser
|
|
109
|
+
redirect to the IdP, code exchange, session/cookie issuance, and CSRF/
|
|
110
|
+
state handling. The existing `validate_oidc_token()` already validates the
|
|
111
|
+
resulting JWT, so this is the smaller of the two.
|
|
112
|
+
- SAML 2.0 (still common in large enterprises): SP metadata, ACS endpoint,
|
|
113
|
+
XML signature validation, NameID/attribute mapping, and IdP-initiated and
|
|
114
|
+
SP-initiated flows.
|
|
115
|
+
- Effort: OIDC login flow on top of the existing validator is roughly a few
|
|
116
|
+
weeks (1 engineer) including session management and tests. Full SAML 2.0 is
|
|
117
|
+
larger, on the order of 1 to 2 months, because XML signature handling and
|
|
118
|
+
multi-IdP quirks are involved; using a vetted SAML library is mandatory
|
|
119
|
+
rather than hand-rolling.
|
|
120
|
+
- Needs: a real IdP to test against (Okta, Azure AD/Entra ID, or Auth0; a
|
|
121
|
+
free developer tenant works for early dev). Integration tests must run
|
|
122
|
+
against that IdP, not mocks alone.
|
|
123
|
+
|
|
124
|
+
### SCIM user/group provisioning
|
|
125
|
+
|
|
126
|
+
- Scope: a SCIM 2.0 server so an IdP can create/update/deactivate users and
|
|
127
|
+
push group membership automatically, instead of users being created on
|
|
128
|
+
first login. Requires `/scim/v2/Users` and `/scim/v2/Groups` endpoints,
|
|
129
|
+
filtering, pagination, PATCH semantics, and mapping SCIM groups onto Loki
|
|
130
|
+
roles/tenants.
|
|
131
|
+
- Effort: roughly 1 to 1.5 months (1 engineer) for a conformant subset plus a
|
|
132
|
+
persistence model for provisioned identities (today there is no durable
|
|
133
|
+
user store; OIDC users are derived per-request from claims).
|
|
134
|
+
- Needs: a SCIM-capable IdP to drive the provisioning (Okta or Azure AD), plus
|
|
135
|
+
a SCIM conformance test harness. This depends on a real user-store model
|
|
136
|
+
landing first.
|
|
137
|
+
|
|
138
|
+
### App-level / tenant RBAC
|
|
139
|
+
|
|
140
|
+
- Scope: roles and policy that go beyond the current four global roles and
|
|
141
|
+
the `read`/`control` scope hierarchy. Concretely: roles scoped per tenant,
|
|
142
|
+
per-tenant policy administration, custom roles, resource-level permissions,
|
|
143
|
+
and an admin surface to manage them.
|
|
144
|
+
- Current state to build on: `Tenant`/`Project` models exist and a
|
|
145
|
+
`tenant:<id>` scope pins a token to one tenant (data isolation). What is
|
|
146
|
+
missing: per-tenant role assignment, a policy model richer than the global
|
|
147
|
+
scope hierarchy, custom/role-definition management, and any UI for it.
|
|
148
|
+
- Effort: roughly 1 to 2 months (1 to 2 engineers) depending on how much
|
|
149
|
+
policy flexibility is committed to (a fixed per-tenant role set is the
|
|
150
|
+
smaller end; arbitrary custom roles and resource-level rules is the larger
|
|
151
|
+
end).
|
|
152
|
+
- Needs: no external IdP, but it pairs naturally with SCIM (group-to-role
|
|
153
|
+
mapping) and benefits from the durable user store noted above.
|
|
154
|
+
|
|
155
|
+
### SOC2
|
|
156
|
+
|
|
157
|
+
- Scope: SOC2 is a compliance PROGRAM, not a code feature. It spans durable,
|
|
158
|
+
tamper-evident audit logging with retention and SIEM export, access reviews,
|
|
159
|
+
change-management process, vendor management, security policies, and a Type
|
|
160
|
+
II observation window (commonly 6 to 12 months of evidence collected under
|
|
161
|
+
an auditor).
|
|
162
|
+
- What exists today that helps but does not constitute SOC2: hash-chained
|
|
163
|
+
audit logging and syslog/SIEM forwarding (see the audit-logging docs). These
|
|
164
|
+
are inputs to an audit, not the certification.
|
|
165
|
+
- Effort: multi-quarter organizational work, typically 6 to 12 months elapsed,
|
|
166
|
+
involving engineering, security, and an external auditor. It is not a sprint
|
|
167
|
+
and should never be represented as a shippable feature.
|
|
168
|
+
- Needs: an auditor engagement, an evidence/observation window, and
|
|
169
|
+
organizational process, in parallel with (not blocking) the code items above.
|
|
170
|
+
|
|
171
|
+
## 3. Sequencing recommendation
|
|
172
|
+
|
|
173
|
+
1. OIDC authorization-code login flow first. It reuses the existing
|
|
174
|
+
`validate_oidc_token()` validator, is the smallest increment, and unlocks
|
|
175
|
+
the most common "we need SSO" enterprise requirement with the least new
|
|
176
|
+
surface. This is the natural next code step after today's OIDC bearer
|
|
177
|
+
validation.
|
|
178
|
+
2. App-level / tenant RBAC next, building on the existing `Tenant` model and
|
|
179
|
+
`tenant:<id>` scope. Most mid-market and enterprise deals ask for "roles
|
|
180
|
+
and tenant isolation" once SSO is in place.
|
|
181
|
+
3. SCIM after a durable user store exists (it depends on one). SCIM is a
|
|
182
|
+
larger-enterprise ask and is most valuable once SSO and RBAC are stable.
|
|
183
|
+
4. SAML in parallel with or after the OIDC login flow, only when a target
|
|
184
|
+
account specifically requires SAML rather than OIDC. Many enterprises
|
|
185
|
+
accept OIDC, so do not build SAML speculatively.
|
|
186
|
+
5. SOC2 runs as parallel organizational work, started early because of the
|
|
187
|
+
observation window, but tracked separately from the code roadmap. The
|
|
188
|
+
existing audit-log foundation is a head start, not a finish line.
|
|
189
|
+
|
|
190
|
+
Rationale: OIDC login -> tenant RBAC -> SCIM is the path that unlocks the most
|
|
191
|
+
enterprise deals soonest while reusing the most existing code. SAML is
|
|
192
|
+
demand-gated. SOC2 is long-lead org work that should begin in parallel rather
|
|
193
|
+
than block feature delivery.
|
|
194
|
+
|
|
195
|
+
## 4. What we do NOT claim
|
|
196
|
+
|
|
197
|
+
Loki Mode does NOT today ship: a browser SSO login flow, SAML support, SCIM
|
|
198
|
+
provisioning, app-level or per-tenant RBAC beyond the global four-role / read-
|
|
199
|
+
control scope model and the `tenant:<id>` data-isolation scope, or SOC2
|
|
200
|
+
certification. The shipped identity surface is: opt-in token auth, opt-in OIDC
|
|
201
|
+
bearer-token validation, a read/control scope model with four predefined
|
|
202
|
+
roles, dashboard WebSocket and REST auth gating, webhook HMAC verification,
|
|
203
|
+
and tenant data isolation via a token scope. Workload-identity OIDC (IRSA /
|
|
204
|
+
Workload Identity) and Kubernetes RBAC/NetworkPolicy are infrastructure
|
|
205
|
+
controls and are not end-user SSO or product RBAC. Any statement beyond this
|
|
206
|
+
is roadmap, not a current capability.
|
package/docs/INSTALLATION.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
The flagship product of [Autonomi](https://www.autonomi.dev/). Loki Mode is a spec-driven autonomous builder with a built-in trust layer that takes any spec to a deployed product and verifies completion with evidence (quality gates plus a completion council), not just a "done" claim. Complete installation instructions for all platforms and use cases.
|
|
4
4
|
|
|
5
|
-
**Version:** v7.
|
|
5
|
+
**Version:** v7.80.1
|
|
6
6
|
|
|
7
7
|
---
|
|
8
8
|
|
|
@@ -395,7 +395,7 @@ provider works inside the container. Provide auth with your Anthropic API key:
|
|
|
395
395
|
# Run Loki Mode in Docker (Claude provider, API-key auth)
|
|
396
396
|
docker run --rm -e ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
|
|
397
397
|
-v $(pwd):/workspace -w /workspace \
|
|
398
|
-
asklokesh/loki-mode:7.
|
|
398
|
+
asklokesh/loki-mode:7.80.1 start ./my-spec.md
|
|
399
399
|
```
|
|
400
400
|
|
|
401
401
|
##### docker compose + .env (no host install)
|
package/loki-ts/dist/loki.js
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
// @bun
|
|
2
|
-
var QQ=Object.defineProperty;var ZQ=($)=>$;function zQ($,Q){this[$]=ZQ.bind(null,Q)}var b=($,Q)=>{for(var Z in Q)QQ($,Z,{get:Q[Z],enumerable:!0,configurable:!0,set:zQ.bind(Q,Z)})};var L=($,Q)=>()=>($&&(Q=$($=0)),Q);var q$=import.meta.require;var h1={};b(h1,{lokiDir:()=>P,homeLokiDir:()=>i$,findRepoRootForVersion:()=>t$,REPO_ROOT:()=>g});import{resolve as a,dirname as r$}from"path";import{fileURLToPath as XQ}from"url";import{existsSync as R$}from"fs";import{homedir as KQ}from"os";function qQ(){let $=b1;for(let Q=0;Q<6;Q++){if(R$(a($,"VERSION"))&&R$(a($,"autonomy/run.sh")))return $;let Z=r$($);if(Z===$)break;$=Z}return a(b1,"..","..","..")}function t$($){let Q=$;for(let Z=0;Z<6;Z++){if(R$(a(Q,"VERSION"))&&R$(a(Q,"autonomy/run.sh")))return Q;let z=r$(Q);if(z===Q)break;Q=z}return a($,"..","..","..")}function P(){return process.env.LOKI_DIR??a(process.cwd(),".loki")}function i$(){return a(KQ(),".loki")}var b1,g;var C=L(()=>{b1=r$(XQ(import.meta.url));g=qQ()});import{readFileSync as VQ}from"fs";import{resolve as JQ,dirname as UQ}from"path";import{fileURLToPath as WQ}from"url";function E$(){if(Q$!==null)return Q$;let $="7.
|
|
2
|
+
var QQ=Object.defineProperty;var ZQ=($)=>$;function zQ($,Q){this[$]=ZQ.bind(null,Q)}var b=($,Q)=>{for(var Z in Q)QQ($,Z,{get:Q[Z],enumerable:!0,configurable:!0,set:zQ.bind(Q,Z)})};var L=($,Q)=>()=>($&&(Q=$($=0)),Q);var q$=import.meta.require;var h1={};b(h1,{lokiDir:()=>P,homeLokiDir:()=>i$,findRepoRootForVersion:()=>t$,REPO_ROOT:()=>g});import{resolve as a,dirname as r$}from"path";import{fileURLToPath as XQ}from"url";import{existsSync as R$}from"fs";import{homedir as KQ}from"os";function qQ(){let $=b1;for(let Q=0;Q<6;Q++){if(R$(a($,"VERSION"))&&R$(a($,"autonomy/run.sh")))return $;let Z=r$($);if(Z===$)break;$=Z}return a(b1,"..","..","..")}function t$($){let Q=$;for(let Z=0;Z<6;Z++){if(R$(a(Q,"VERSION"))&&R$(a(Q,"autonomy/run.sh")))return Q;let z=r$(Q);if(z===Q)break;Q=z}return a($,"..","..","..")}function P(){return process.env.LOKI_DIR??a(process.cwd(),".loki")}function i$(){return a(KQ(),".loki")}var b1,g;var C=L(()=>{b1=r$(XQ(import.meta.url));g=qQ()});import{readFileSync as VQ}from"fs";import{resolve as JQ,dirname as UQ}from"path";import{fileURLToPath as WQ}from"url";function E$(){if(Q$!==null)return Q$;let $="7.80.1";if(typeof $==="string"&&$.length>0)return Q$=$,Q$;try{let Q=UQ(WQ(import.meta.url)),Z=t$(Q);Q$=VQ(JQ(Z,"VERSION"),"utf-8").trim()}catch{Q$="unknown"}return Q$}var Q$=null;var e$=L(()=>{C()});var g1={};b(g1,{runOrThrow:()=>HQ,run:()=>F,readStreamCapped:()=>m1,commandVersion:()=>BQ,commandExists:()=>f,ShellError:()=>$1,MAX_STDOUT_BYTES:()=>v1});async function m1($,Q=v1){let Z=$.getReader(),z=new TextDecoder,X="",q=0;try{while(q<Q){let{done:K,value:U}=await Z.read();if(K)break;if(!U)continue;if(q+=U.byteLength,q>Q){let J=U.byteLength-(q-Q);X+=z.decode(U.subarray(0,J),{stream:!0});break}X+=z.decode(U,{stream:!0})}X+=z.decode()}finally{try{await Z.cancel()}catch{}Z.releaseLock()}return X}async function F($,Q={}){let Z=Bun.spawn({cmd:[...$],stdout:"pipe",stderr:"pipe",env:Q.env?{...process.env,...Q.env}:process.env,cwd:Q.cwd}),z,X;if(Q.timeoutMs&&Q.timeoutMs>0)z=setTimeout(()=>{try{Z.kill("SIGTERM")}catch{}X=setTimeout(()=>{try{Z.kill("SIGKILL")}catch{}},2000)},Q.timeoutMs);try{let[q,K,U]=await Promise.all([m1(Z.stdout),new Response(Z.stderr).text(),Z.exited]);return{stdout:q,stderr:K,exitCode:U}}finally{if(z)clearTimeout(z);if(X)clearTimeout(X)}}async function HQ($,Q={}){let Z=await F($,Q);if(Z.exitCode!==0)throw new $1(`command failed (${Z.exitCode}): ${$.join(" ")}`,Z.exitCode,Z.stdout,Z.stderr);return Z}async function f($){let Q=GQ($),Z=await F(["sh","-c",`command -v ${Q}`],{timeoutMs:5000});if(Z.exitCode===0)return Z.stdout.trim()||null;return null}function GQ($){if(!/^[A-Za-z0-9._/-]+$/.test($))throw Error(`refused to shell-escape suspect token: ${$}`);return $}async function BQ($,Q="--version"){if(!await f($))return null;let z=await F([$,Q],{timeoutMs:5000});if(z.exitCode!==0)return null;return((z.stdout||z.stderr).split(/\r?\n/)[0]?.trim()??"")||null}var v1=16777216,$1;var d=L(()=>{$1=class $1 extends Error{message;exitCode;stdout;stderr;constructor($,Q,Z,z){super($);this.message=$;this.exitCode=Q;this.stdout=Z;this.stderr=z;this.name="ShellError"}}});function s($){return YQ?"":$}var YQ,O,S,_,_Z,I,k,h,V;var c=L(()=>{YQ=(process.env.NO_COLOR??"").length>0;O=s("\x1B[0;31m"),S=s("\x1B[0;32m"),_=s("\x1B[1;33m"),_Z=s("\x1B[0;34m"),I=s("\x1B[0;36m"),k=s("\x1B[1m"),h=s("\x1B[2m"),V=s("\x1B[0m")});import{existsSync as jQ}from"fs";async function Z$(){if(Y$!==void 0)return Y$;let $="/opt/homebrew/bin/python3.12";if(jQ($))return Y$=$,$;let Q=await f("python3.12");if(Q)return Y$=Q,Q;let Z=await f("python3");return Y$=Z,Z}async function z$($,Q={}){let Z=await Z$();if(!Z)return{stdout:"",stderr:"python3 not found",exitCode:127};return F([Z,"-c",$],Q)}var Y$;var V$=L(()=>{d()});var X0={};b(X0,{runStatus:()=>oQ});import{existsSync as y,readFileSync as U$,readdirSync as r1,statSync as t1}from"fs";import{resolve as D,basename as vQ}from"path";import{homedir as mQ}from"os";function i1($){let Q=Math.trunc($);if(Q>=1e6)return`${(Math.trunc(Q/1e6*10)/10).toFixed(1)}M`;if(Q>=1000)return`${(Math.trunc(Q/1000*10)/10).toFixed(1)}K`;return String(Q)}function e1($,Q,Z){if(Q===0)return null;let z=Math.trunc($*100/Q),X=Math.trunc($*N$/Q);if(X>N$)X=N$;let q=N$-X,K=S;if(z>=80)K=O;else if(z>=50)K=_;let U="=".repeat(Math.max(0,X))+" ".repeat(Math.max(0,q)),J=i1($),W=i1(Q);return` ${k}${Z}${V} ${K}[${U}]${V} ${z}% (${J} / ${W})`}async function fQ(){if(await f("jq"))return!0;return process.stdout.write(`${O}Error: jq is required but not installed.${V}
|
|
3
3
|
`),process.stdout.write(`Install with:
|
|
4
4
|
`),process.stdout.write(` brew install jq (macOS)
|
|
5
5
|
`),process.stdout.write(` apt install jq (Debian/Ubuntu)
|
|
@@ -793,4 +793,4 @@ Set LOKI_LEGACY_BASH=1 to force the bash CLI for every command.
|
|
|
793
793
|
`),2}default:return process.stderr.write(`Unknown command: ${Q}
|
|
794
794
|
`),process.stderr.write($Q),2}}s1();process.on("SIGINT",()=>process.exit(130));process.on("SIGTERM",()=>process.exit(143));var qZ=await KZ(Bun.argv.slice(2));process.exit(qZ);
|
|
795
795
|
|
|
796
|
-
//# debugId=
|
|
796
|
+
//# debugId=7F54054FBBA43A1E64756E2164756E21
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
"""
|
|
2
|
+
LokiStore: pluggable, local-first storage abstraction for Loki Mode.
|
|
3
|
+
|
|
4
|
+
This package is the consolidation spine for durable artifacts (state,
|
|
5
|
+
checkpoints, memory blobs, healing artifacts, etc.). Features should bind to
|
|
6
|
+
this abstraction instead of hardcoding `.loki/` paths or embedding their own
|
|
7
|
+
cloud SDK clients.
|
|
8
|
+
|
|
9
|
+
Design goals
|
|
10
|
+
------------
|
|
11
|
+
- Local-first by default. With nothing configured, LocalStore writes to the
|
|
12
|
+
project `.loki/` directory with byte-identical behavior to today's direct
|
|
13
|
+
file writes (atomic temp+rename, fcntl advisory locking, path-traversal
|
|
14
|
+
guard). Zero new dependencies, zero behavior change for local users.
|
|
15
|
+
- Optional cloud backends (S3, GCS, Azure Blob) behind lazy imports. The
|
|
16
|
+
cloud SDK is imported ONLY inside the backend's __init__, so a missing
|
|
17
|
+
dependency raises a clear, actionable error ONLY when that backend is
|
|
18
|
+
explicitly selected -- never at import time, never for local users.
|
|
19
|
+
- One interface (put/get/get_to/exists/list/delete) so callers are backend
|
|
20
|
+
agnostic.
|
|
21
|
+
|
|
22
|
+
Quick start
|
|
23
|
+
-----------
|
|
24
|
+
from lokistore import get_store
|
|
25
|
+
|
|
26
|
+
store = get_store() # local by default, honors LOKI_DIR/TARGET_DIR
|
|
27
|
+
store.put("state/checkpoints/cp-1/metadata.json", b"{...}")
|
|
28
|
+
data = store.get("state/checkpoints/cp-1/metadata.json")
|
|
29
|
+
for key in store.list("state/checkpoints/"):
|
|
30
|
+
...
|
|
31
|
+
|
|
32
|
+
Configuration (env or config dict)
|
|
33
|
+
----------------------------------
|
|
34
|
+
- LOKI_STORAGE_BACKEND : local | s3 | gcs | azure-blob (default: local)
|
|
35
|
+
- LOKI_STORAGE_BUCKET : bucket / container name (cloud backends)
|
|
36
|
+
- LOKI_STORAGE_PREFIX : key prefix within the bucket (optional)
|
|
37
|
+
- LOKI_STORAGE_REGION : region (s3, optional)
|
|
38
|
+
|
|
39
|
+
The metadata backend (sqlite default, postgres later) is selected separately;
|
|
40
|
+
see get_metadata_backend() / METADATA notes below. For this release the blob
|
|
41
|
+
backends are the core deliverable and the metadata selector is a thin,
|
|
42
|
+
documented stub that defaults to the existing sqlite path.
|
|
43
|
+
"""
|
|
44
|
+
|
|
45
|
+
from .base import LokiStore, StoreError, BackendNotAvailableError
|
|
46
|
+
from .local import LocalStore
|
|
47
|
+
from .factory import (
|
|
48
|
+
get_store,
|
|
49
|
+
build_store,
|
|
50
|
+
get_metadata_backend,
|
|
51
|
+
resolve_local_base,
|
|
52
|
+
)
|
|
53
|
+
|
|
54
|
+
__all__ = [
|
|
55
|
+
"LokiStore",
|
|
56
|
+
"StoreError",
|
|
57
|
+
"BackendNotAvailableError",
|
|
58
|
+
"LocalStore",
|
|
59
|
+
"get_store",
|
|
60
|
+
"build_store",
|
|
61
|
+
"get_metadata_backend",
|
|
62
|
+
"resolve_local_base",
|
|
63
|
+
]
|
|
64
|
+
|
|
65
|
+
__version__ = "1.0.0"
|
|
@@ -0,0 +1,172 @@
|
|
|
1
|
+
"""
|
|
2
|
+
LokiStore interface and shared helpers.
|
|
3
|
+
|
|
4
|
+
Defines the abstract blob-store contract that every backend implements, plus
|
|
5
|
+
errors and a key-normalization helper shared across backends.
|
|
6
|
+
"""
|
|
7
|
+
|
|
8
|
+
from __future__ import annotations
|
|
9
|
+
|
|
10
|
+
import os
|
|
11
|
+
from abc import ABC, abstractmethod
|
|
12
|
+
from typing import List, Union
|
|
13
|
+
|
|
14
|
+
|
|
15
|
+
class StoreError(Exception):
|
|
16
|
+
"""Base class for LokiStore errors."""
|
|
17
|
+
|
|
18
|
+
|
|
19
|
+
class BackendNotAvailableError(StoreError):
|
|
20
|
+
"""
|
|
21
|
+
Raised when a cloud backend is explicitly selected but its SDK (or
|
|
22
|
+
required configuration) is not available. The message names the missing
|
|
23
|
+
package and how to install it so the failure is actionable.
|
|
24
|
+
"""
|
|
25
|
+
|
|
26
|
+
|
|
27
|
+
def normalize_key(key: str) -> str:
|
|
28
|
+
"""
|
|
29
|
+
Normalize a store key to a clean POSIX-style relative path.
|
|
30
|
+
|
|
31
|
+
Keys are relative paths like "state/checkpoints/cp-1/metadata.json".
|
|
32
|
+
This rejects absolute paths and any key that would traverse outside the
|
|
33
|
+
store root (a leading or embedded ".." segment). The same guard is used
|
|
34
|
+
by every backend so traversal is rejected uniformly, whether the bytes
|
|
35
|
+
land on a local filesystem or in an object-store key namespace.
|
|
36
|
+
|
|
37
|
+
Args:
|
|
38
|
+
key: A POSIX-style relative path.
|
|
39
|
+
|
|
40
|
+
Returns:
|
|
41
|
+
The normalized key (forward slashes, no leading slash, no "."/".."
|
|
42
|
+
segments collapsed away).
|
|
43
|
+
|
|
44
|
+
Raises:
|
|
45
|
+
ValueError: If the key is empty, absolute, or escapes the root.
|
|
46
|
+
"""
|
|
47
|
+
if not isinstance(key, str) or not key:
|
|
48
|
+
raise ValueError("store key must be a non-empty string")
|
|
49
|
+
|
|
50
|
+
# Treat backslashes as separators too, so a Windows-style key cannot
|
|
51
|
+
# smuggle a traversal past the split() check below.
|
|
52
|
+
candidate = key.replace("\\", "/")
|
|
53
|
+
|
|
54
|
+
if candidate.startswith("/"):
|
|
55
|
+
raise ValueError(f"absolute keys are not allowed: {key!r}")
|
|
56
|
+
|
|
57
|
+
parts: List[str] = []
|
|
58
|
+
for segment in candidate.split("/"):
|
|
59
|
+
if segment == "" or segment == ".":
|
|
60
|
+
# Collapse empty (e.g. "a//b") and current-dir segments.
|
|
61
|
+
continue
|
|
62
|
+
if segment == "..":
|
|
63
|
+
raise ValueError(f"path traversal is not allowed in key: {key!r}")
|
|
64
|
+
parts.append(segment)
|
|
65
|
+
|
|
66
|
+
if not parts:
|
|
67
|
+
raise ValueError(f"key resolves to an empty path: {key!r}")
|
|
68
|
+
|
|
69
|
+
return "/".join(parts)
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
def read_source_bytes(data: Union[bytes, bytearray, str, os.PathLike]) -> bytes:
|
|
73
|
+
"""
|
|
74
|
+
Coerce a put() source argument into bytes.
|
|
75
|
+
|
|
76
|
+
Accepts raw bytes (returned as-is) or a path-like pointing at a file whose
|
|
77
|
+
contents are read in binary. A plain str is treated as a FILESYSTEM PATH,
|
|
78
|
+
not as text, to match the put(key, bytes_or_path) contract; callers that
|
|
79
|
+
want to store a string must encode it themselves.
|
|
80
|
+
|
|
81
|
+
Args:
|
|
82
|
+
data: bytes/bytearray, or a path-like to an existing file.
|
|
83
|
+
|
|
84
|
+
Returns:
|
|
85
|
+
The bytes to store.
|
|
86
|
+
|
|
87
|
+
Raises:
|
|
88
|
+
FileNotFoundError: If a path is given and the file does not exist.
|
|
89
|
+
TypeError: If the argument is neither bytes nor a path-like.
|
|
90
|
+
"""
|
|
91
|
+
if isinstance(data, (bytes, bytearray)):
|
|
92
|
+
return bytes(data)
|
|
93
|
+
if isinstance(data, (str, os.PathLike)):
|
|
94
|
+
with open(data, "rb") as f:
|
|
95
|
+
return f.read()
|
|
96
|
+
raise TypeError(
|
|
97
|
+
"put() source must be bytes or a path-like to a file, "
|
|
98
|
+
f"got {type(data).__name__}"
|
|
99
|
+
)
|
|
100
|
+
|
|
101
|
+
|
|
102
|
+
class LokiStore(ABC):
|
|
103
|
+
"""
|
|
104
|
+
Abstract pluggable blob store.
|
|
105
|
+
|
|
106
|
+
All keys are POSIX-style relative paths (see normalize_key). Implementations
|
|
107
|
+
must apply normalize_key to every incoming key so traversal is rejected and
|
|
108
|
+
behavior is uniform across backends.
|
|
109
|
+
|
|
110
|
+
The contract is intentionally small (no streaming, no multipart, no
|
|
111
|
+
credential framework) so backends stay thin. Larger payload handling can be
|
|
112
|
+
layered on later without changing this interface.
|
|
113
|
+
"""
|
|
114
|
+
|
|
115
|
+
@abstractmethod
|
|
116
|
+
def put(self, key: str, data: Union[bytes, bytearray, str, os.PathLike]) -> None:
|
|
117
|
+
"""
|
|
118
|
+
Store bytes under key. The source is either raw bytes or a path-like
|
|
119
|
+
to a file whose contents are stored. Overwrites any existing value
|
|
120
|
+
atomically where the backend supports it.
|
|
121
|
+
"""
|
|
122
|
+
|
|
123
|
+
@abstractmethod
|
|
124
|
+
def get(self, key: str) -> bytes:
|
|
125
|
+
"""
|
|
126
|
+
Return the bytes stored under key.
|
|
127
|
+
|
|
128
|
+
Raises:
|
|
129
|
+
FileNotFoundError: If the key does not exist.
|
|
130
|
+
"""
|
|
131
|
+
|
|
132
|
+
@abstractmethod
|
|
133
|
+
def get_to(self, key: str, dest_path: Union[str, os.PathLike]) -> None:
|
|
134
|
+
"""
|
|
135
|
+
Write the bytes stored under key to dest_path (a local filesystem
|
|
136
|
+
path). Parent directories are created as needed.
|
|
137
|
+
|
|
138
|
+
Raises:
|
|
139
|
+
FileNotFoundError: If the key does not exist.
|
|
140
|
+
"""
|
|
141
|
+
|
|
142
|
+
@abstractmethod
|
|
143
|
+
def exists(self, key: str) -> bool:
|
|
144
|
+
"""Return True if key exists in the store."""
|
|
145
|
+
|
|
146
|
+
@abstractmethod
|
|
147
|
+
def list(self, prefix: str = "") -> List[str]:
|
|
148
|
+
"""
|
|
149
|
+
List keys under prefix (a POSIX-style relative PATH prefix). An empty
|
|
150
|
+
prefix lists every key. Returns normalized keys, sorted, relative to
|
|
151
|
+
the store root.
|
|
152
|
+
|
|
153
|
+
Prefix semantics (portable contract):
|
|
154
|
+
- prefix is a PATH prefix matched at slash boundaries, NOT a raw
|
|
155
|
+
substring. A slash-terminated prefix like "state/" selects every key
|
|
156
|
+
beneath the "state" directory and is the portable form that behaves
|
|
157
|
+
IDENTICALLY across all backends (local and cloud).
|
|
158
|
+
- An empty prefix lists everything.
|
|
159
|
+
|
|
160
|
+
A non-slash-terminated prefix (e.g. "state") is normalized to its
|
|
161
|
+
directory form before matching, so it selects keys under "state/" and
|
|
162
|
+
does NOT match siblings that merely share the leading characters (e.g.
|
|
163
|
+
"stateful/x.json"). Callers wanting portable results should pass a
|
|
164
|
+
slash-terminated prefix.
|
|
165
|
+
"""
|
|
166
|
+
|
|
167
|
+
@abstractmethod
|
|
168
|
+
def delete(self, key: str) -> bool:
|
|
169
|
+
"""
|
|
170
|
+
Delete key. Returns True if a value was removed, False if the key did
|
|
171
|
+
not exist (delete is idempotent and never raises on a missing key).
|
|
172
|
+
"""
|