@delma/fylo 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.env.example +16 -0
- package/.github/copilot-instructions.md +113 -0
- package/.github/prompts/issue.prompt.md +19 -0
- package/.github/prompts/pr.prompt.md +18 -0
- package/.github/prompts/release.prompt.md +49 -0
- package/.github/prompts/review-pr.prompt.md +19 -0
- package/.github/prompts/sync-main.prompt.md +14 -0
- package/.github/workflows/ci.yml +37 -0
- package/.github/workflows/publish.yml +101 -0
- package/.prettierrc +7 -0
- package/LICENSE +21 -0
- package/README.md +230 -0
- package/eslint.config.js +28 -0
- package/package.json +51 -0
- package/src/CLI +37 -0
- package/src/adapters/cipher.ts +174 -0
- package/src/adapters/redis.ts +71 -0
- package/src/adapters/s3.ts +67 -0
- package/src/core/directory.ts +418 -0
- package/src/core/extensions.ts +19 -0
- package/src/core/format.ts +486 -0
- package/src/core/parser.ts +876 -0
- package/src/core/query.ts +48 -0
- package/src/core/walker.ts +167 -0
- package/src/index.ts +1088 -0
- package/src/types/fylo.d.ts +139 -0
- package/src/types/index.d.ts +3 -0
- package/src/types/query.d.ts +73 -0
- package/tests/collection/truncate.test.ts +56 -0
- package/tests/data.ts +110 -0
- package/tests/index.ts +19 -0
- package/tests/integration/create.test.ts +57 -0
- package/tests/integration/delete.test.ts +147 -0
- package/tests/integration/edge-cases.test.ts +232 -0
- package/tests/integration/encryption.test.ts +176 -0
- package/tests/integration/export.test.ts +61 -0
- package/tests/integration/join-modes.test.ts +221 -0
- package/tests/integration/nested.test.ts +212 -0
- package/tests/integration/operators.test.ts +167 -0
- package/tests/integration/read.test.ts +203 -0
- package/tests/integration/rollback.test.ts +105 -0
- package/tests/integration/update.test.ts +130 -0
- package/tests/mocks/cipher.ts +55 -0
- package/tests/mocks/redis.ts +13 -0
- package/tests/mocks/s3.ts +114 -0
- package/tests/schemas/album.d.ts +5 -0
- package/tests/schemas/album.json +5 -0
- package/tests/schemas/comment.d.ts +7 -0
- package/tests/schemas/comment.json +7 -0
- package/tests/schemas/photo.d.ts +7 -0
- package/tests/schemas/photo.json +7 -0
- package/tests/schemas/post.d.ts +6 -0
- package/tests/schemas/post.json +6 -0
- package/tests/schemas/tip.d.ts +7 -0
- package/tests/schemas/tip.json +7 -0
- package/tests/schemas/todo.d.ts +6 -0
- package/tests/schemas/todo.json +6 -0
- package/tests/schemas/user.d.ts +23 -0
- package/tests/schemas/user.json +23 -0
- package/tsconfig.json +19 -0
package/.env.example
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
LOGGING=
|
|
2
|
+
STRICT=
|
|
3
|
+
S3_REGION=ca-central-1
|
|
4
|
+
S3_ACCESS_KEY_ID=HELLO
|
|
5
|
+
S3_SECRET_ACCESS_KEY=WORLD
|
|
6
|
+
S3_ENDPOINT=https//example.com
|
|
7
|
+
BUCKET_PREFIX="byos-test"
|
|
8
|
+
SCHEMA_DIR=/path/to/schema/dir
|
|
9
|
+
REDIS_URL=redis://localhost:6379
|
|
10
|
+
REDIS_CONN_TIMEOUT=5000
|
|
11
|
+
REDIS_IDLE_TIMEOUT=30000
|
|
12
|
+
REDIS_AUTO_RECONNECT=
|
|
13
|
+
REDIS_MAX_RETRIES=10
|
|
14
|
+
REDIS_ENABLE_OFFLINE_QUEUE=
|
|
15
|
+
REDIS_ENABLE_AUTO_PIPELINING=
|
|
16
|
+
REDIS_TLS=
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
# FYLO — Project Guidelines
|
|
2
|
+
|
|
3
|
+
## Overview
|
|
4
|
+
|
|
5
|
+
FYLO (`@vyckr/fylo`) is an S3-backed NoSQL document store with SQL parsing, Redis pub/sub for real-time events, and a CLI. Documents are stored as S3 key paths — not as file contents — with dual key layouts for data access and indexed queries.
|
|
6
|
+
|
|
7
|
+
**Assume a serverless deployment model** (e.g., AWS Lambda, Cloudflare Workers). This means:
|
|
8
|
+
- No persistent in-memory state across invocations — every request starts cold
|
|
9
|
+
- Distributed coordination (e.g., TTID uniqueness) must use external stores like Redis, not in-process caches
|
|
10
|
+
- Avoid long-lived connections, background threads, or singleton patterns that assume process longevity
|
|
11
|
+
- Keep cold-start overhead minimal — lazy initialization over eager setup
|
|
12
|
+
|
|
13
|
+
## Architecture
|
|
14
|
+
|
|
15
|
+
### Key Storage Format
|
|
16
|
+
|
|
17
|
+
- **Data keys**: `{ttid}/{field}/{value}` — keyed by document ID for full-doc retrieval
|
|
18
|
+
- **Index keys**: `{field}/{value}/{ttid}` — keyed by field for query lookups
|
|
19
|
+
- Nested objects flatten to path segments: `address/city/Toronto`
|
|
20
|
+
- Forward slashes in values are escaped with an ASCII substitute
|
|
21
|
+
|
|
22
|
+
### Core Modules
|
|
23
|
+
|
|
24
|
+
| Module | Responsibility |
|
|
25
|
+
|--------|---------------|
|
|
26
|
+
| `src/index.ts` | Main `Fylo` class — CRUD, SQL execution, joins, bulk ops |
|
|
27
|
+
| `src/core/parser.ts` | SQL lexer/parser — tokenizes SQL into query objects |
|
|
28
|
+
| `src/core/query.ts` | Converts `$ops` into glob patterns for S3 key matching |
|
|
29
|
+
| `src/core/walker.ts` | S3 key traversal, document data retrieval, Redis event streaming |
|
|
30
|
+
| `src/core/directory.ts` | Key extraction, reconstruction, rollback tracking |
|
|
31
|
+
| `src/core/format.ts` | Console formatting for query output |
|
|
32
|
+
| `src/adapters/s3.ts` | S3 adapter (Bun S3Client) |
|
|
33
|
+
| `src/adapters/redis.ts` | Redis adapter (Bun RedisClient) |
|
|
34
|
+
| `src/cli/index.ts` | CLI entry point (`fylo.query`) |
|
|
35
|
+
|
|
36
|
+
### Folder Structure
|
|
37
|
+
|
|
38
|
+
```
|
|
39
|
+
src/
|
|
40
|
+
index.ts # Public API — main Fylo class
|
|
41
|
+
adapters/ # I/O boundary abstractions (S3, Redis)
|
|
42
|
+
core/ # Internal domain logic (parser, query, walker, directory)
|
|
43
|
+
cli/ # CLI entry point
|
|
44
|
+
types/ # Type declarations (.d.ts only — separate from implementation)
|
|
45
|
+
tests/
|
|
46
|
+
data.ts # Shared test data URLs
|
|
47
|
+
index.ts # Test barrel
|
|
48
|
+
mocks/ # Mock adapters (S3, Redis) for testing
|
|
49
|
+
schemas/ # CHEX-generated test schemas (.d.ts + .json)
|
|
50
|
+
integration/ # End-to-end tests (CRUD, operators, joins, edge cases)
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
### Dependencies
|
|
54
|
+
|
|
55
|
+
- **`@vyckr/ttid`** — Time-based unique ID system. `TTID.generate()` creates new IDs; `TTID.generate(existingId)` creates a versioned ID sharing the same creation-time prefix.
|
|
56
|
+
- **`@vyckr/chex`** — Schema validation. Generates `interface` declarations in `.d.ts` files. Generic constraints must use `Record<string, any>` (not `Record<string, unknown>`) to accept these interfaces.
|
|
57
|
+
- **`Bun.Glob`** — Pattern matching for queries. Does NOT support negation extglob `!(pattern)`. Operators like `$ne`, `$gt`, `$lt` use broad globs with post-filtering instead.
|
|
58
|
+
|
|
59
|
+
## Engineering Standards
|
|
60
|
+
|
|
61
|
+
- **SOLID principles**: Single responsibility per class/method, depend on abstractions (e.g., S3/Redis adapters), open for extension without modifying core logic
|
|
62
|
+
- **Clean code**: Descriptive naming, small focused functions, no dead code or commented-out blocks, DRY without premature abstraction
|
|
63
|
+
- **Test discipline**: When changing `src/` code, update or add corresponding tests in `tests/` — never leave tests stale after a behaviour change
|
|
64
|
+
- **Error handling**: Fail fast with meaningful errors at system boundaries; use rollback mechanisms for partial writes
|
|
65
|
+
- **No magic values**: Use constants or environment variables; avoid hardcoded strings/numbers in logic
|
|
66
|
+
- **Type safety**: Leverage TypeScript's type system fully — avoid `any` in implementation code, prefer narrow types, and validate at I/O boundaries
|
|
67
|
+
|
|
68
|
+
## Code Style
|
|
69
|
+
|
|
70
|
+
- **Runtime**: Bun (ESNext target, ES modules)
|
|
71
|
+
- **Strict TypeScript**: `strict: true`, `noImplicitReturns`, `isolatedModules`
|
|
72
|
+
- **ESLint** enforces `@typescript-eslint/no-explicit-any` in `src/` and `tests/` — use it only in type declarations (`.d.ts`)
|
|
73
|
+
- **No default exports** except the main `Fylo` class
|
|
74
|
+
- Prefer `class` with `static` methods for modules (no standalone functions)
|
|
75
|
+
- Use `_ttid` branded type for document IDs — never plain `string`
|
|
76
|
+
- Prefix internal/test type names with underscore: `_post`, `_album`, `_storeQuery`
|
|
77
|
+
- Type declarations live in `src/types/*.d.ts` — keep separate from implementation
|
|
78
|
+
|
|
79
|
+
## Build & Test
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
bun test # Run all tests
|
|
83
|
+
bun run build # Compile TypeScript
|
|
84
|
+
bun run typecheck # Type-check without emitting
|
|
85
|
+
bun run lint # ESLint
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
- Tests use `bun:test` — `describe`, `test`, `expect`, `mock`, `beforeAll`, `afterAll`
|
|
89
|
+
- S3 and Redis are mocked via `mock.module()` in every test file using `tests/mocks/s3.ts` and `tests/mocks/redis.ts`
|
|
90
|
+
- Test schemas live in `tests/schemas/*.d.ts` as global `interface` declarations (generated by CHEX)
|
|
91
|
+
- Test data URLs are centralized in `tests/data.ts`
|
|
92
|
+
|
|
93
|
+
## Conventions
|
|
94
|
+
|
|
95
|
+
- Collection names may contain hyphens (e.g., `ec-test`, `jm-album`) — the parser supports this
|
|
96
|
+
- Nested field access in SQL uses dot notation (`address.city`) which the parser converts to slash-separated paths (`address/city`)
|
|
97
|
+
- `putData` creates documents; `patchDoc` updates them (deletes old keys, writes new ones)
|
|
98
|
+
- `getDocData` retrieves keys for a specific TTID — filters by exact ID, not just prefix
|
|
99
|
+
- Query `$ops` use OR semantics — a document matches if it satisfies at least one operator
|
|
100
|
+
- `$limit` on queries without `$ops` uses S3 `maxKeys`; with `$ops` it post-filters after glob matching
|
|
101
|
+
|
|
102
|
+
## Environment Variables
|
|
103
|
+
|
|
104
|
+
| Variable | Purpose |
|
|
105
|
+
|----------|---------|
|
|
106
|
+
| `BUCKET_PREFIX` | S3 bucket name prefix |
|
|
107
|
+
| `S3_ACCESS_KEY_ID` / `AWS_ACCESS_KEY_ID` | S3 credentials |
|
|
108
|
+
| `S3_SECRET_ACCESS_KEY` / `AWS_SECRET_ACCESS_KEY` | S3 credentials |
|
|
109
|
+
| `S3_REGION` / `AWS_REGION` | S3 region |
|
|
110
|
+
| `S3_ENDPOINT` / `AWS_ENDPOINT` | S3 endpoint (for compatible stores) |
|
|
111
|
+
| `REDIS_URL` | Redis connection URL |
|
|
112
|
+
| `LOGGING` | Enable debug logging |
|
|
113
|
+
| `STRICT` | Enable schema validation via CHEX |
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: "Create a GitHub issue for a bug or feature request"
|
|
3
|
+
argument-hint: "Describe the bug or feature request"
|
|
4
|
+
agent: "agent"
|
|
5
|
+
tools: [runInTerminal]
|
|
6
|
+
---
|
|
7
|
+
Create a GitHub issue. The user's description is provided as the argument.
|
|
8
|
+
|
|
9
|
+
Determine whether the description sounds like a bug or a feature request, then create the issue with `gh issue create` using the following structure:
|
|
10
|
+
|
|
11
|
+
**For a bug:**
|
|
12
|
+
- Title: "fix: <short description>"
|
|
13
|
+
- Body sections: **Describe the bug**, **Steps to reproduce**, **Expected behaviour**, **Actual behaviour**, **Possible cause** (reference relevant lines in `src/index.ts` if applicable), **Environment** (Bun version, TypeScript version, AWS/S3 endpoint details).
|
|
14
|
+
|
|
15
|
+
**For a feature request:**
|
|
16
|
+
- Title: "feat: <short description>"
|
|
17
|
+
- Body sections: **Problem**, **Proposed solution**, **Alternatives considered**, **Affected API** (list any methods in `src/types/fylo.d.ts` or `src/types/query.d.ts` that would change).
|
|
18
|
+
|
|
19
|
+
Apply the appropriate label (`bug` or `enhancement`) via `--label`. Print the issue URL when done.
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: "Create a pull request for the current branch into main"
|
|
3
|
+
agent: "agent"
|
|
4
|
+
tools: [runInTerminal]
|
|
5
|
+
---
|
|
6
|
+
Create a pull request for the current branch into `main`.
|
|
7
|
+
|
|
8
|
+
1. Run `git status` and stop if there are uncommitted changes — ask the user to commit or stash them first.
|
|
9
|
+
2. Run `git log main..HEAD --oneline` to list commits on this branch.
|
|
10
|
+
3. Run `git diff main...HEAD` to understand all changes.
|
|
11
|
+
4. Push the branch to origin if it has no upstream: `git push -u origin HEAD`.
|
|
12
|
+
5. Create the PR with `gh pr create` using:
|
|
13
|
+
- A concise title (≤ 70 chars) derived from the branch name and commits.
|
|
14
|
+
- A body with three sections:
|
|
15
|
+
- **Summary** — bullet list of what changed and why.
|
|
16
|
+
- **Test plan** — checklist of how to verify the changes (reference `bun test` where relevant).
|
|
17
|
+
- **Breaking changes** — any changes to public API in [src/types/fylo.d.ts](src/types/fylo.d.ts) or [src/types/query.d.ts](src/types/query.d.ts); write "None" if there are none.
|
|
18
|
+
6. Print the PR URL.
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: "Create a release branch, publish to npm via CI, then merge to main"
|
|
3
|
+
agent: "agent"
|
|
4
|
+
tools: [runInTerminal]
|
|
5
|
+
---
|
|
6
|
+
Create a release branch, publish to npm via CI, then merge to main.
|
|
7
|
+
|
|
8
|
+
1. Run `bun test` and stop if any tests fail.
|
|
9
|
+
|
|
10
|
+
2. Determine the new version automatically based on unreleased commits:
|
|
11
|
+
`git log $(git describe --tags --abbrev=0 2>/dev/null || git rev-list --max-parents=0 HEAD)..HEAD --oneline`
|
|
12
|
+
|
|
13
|
+
Apply these rules to select the bump type:
|
|
14
|
+
- **major** — any commit with a `!` breaking-change marker (e.g. `feat!:`, `fix!:`) or a `BREAKING CHANGE` footer.
|
|
15
|
+
- **minor** — one or more `feat:` commits and no breaking changes.
|
|
16
|
+
- **patch** — only `fix:`, `chore:`, `docs:`, `refactor:`, `test:`, or `perf:` commits.
|
|
17
|
+
|
|
18
|
+
Compute the new version by incrementing the corresponding part of the current `"version"` in [package.json](package.json) and resetting lower parts to zero. Show the chosen version and the reasoning to the user before proceeding.
|
|
19
|
+
|
|
20
|
+
3. Update `"version"` in [package.json](package.json) to the new version.
|
|
21
|
+
|
|
22
|
+
4. Fetch the latest main and create a release branch from it:
|
|
23
|
+
```
|
|
24
|
+
git fetch origin main
|
|
25
|
+
git checkout -b release/<version> origin/main
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
5. Stage all changes and commit:
|
|
29
|
+
`git add -A && git commit -m "chore: release v<version>"`
|
|
30
|
+
|
|
31
|
+
6. Push the branch:
|
|
32
|
+
`git push -u origin release/<version>`
|
|
33
|
+
|
|
34
|
+
7. Tell the user that the `publish` workflow will now run on GitHub Actions:
|
|
35
|
+
- It verifies the branch name matches `package.json` version.
|
|
36
|
+
- It runs tests, publishes to npm, creates a git tag, and opens a GitHub release.
|
|
37
|
+
- The NPM_TOKEN secret must be set in repo Settings → Secrets → Actions.
|
|
38
|
+
|
|
39
|
+
8. Once the workflow passes (user confirms), create a PR and merge it to main:
|
|
40
|
+
```
|
|
41
|
+
gh pr create --title "chore: release v<version>" --body "Release v<version>" --base main --head release/<version>
|
|
42
|
+
gh pr merge --merge --delete-branch
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
9. Switch back to main and pull:
|
|
46
|
+
```
|
|
47
|
+
git checkout main
|
|
48
|
+
git pull
|
|
49
|
+
```
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: "Review a pull request by number or URL"
|
|
3
|
+
argument-hint: "PR number or URL (e.g. 42)"
|
|
4
|
+
agent: "agent"
|
|
5
|
+
tools: [runInTerminal]
|
|
6
|
+
---
|
|
7
|
+
Review the pull request given as an argument (PR number or URL).
|
|
8
|
+
|
|
9
|
+
1. Fetch the PR details: `gh pr view <arg> --json title,body,headRefName,baseRefName,files`
|
|
10
|
+
2. Fetch the diff: `gh pr diff <arg>`
|
|
11
|
+
3. Review the changes with focus on:
|
|
12
|
+
- **Correctness** — logic errors, edge cases missed in query parsing ([src/core/parser.ts](src/core/parser.ts)), S3 operations ([src/adapters/s3.ts](src/adapters/s3.ts)), or directory/index management ([src/core/directory.ts](src/core/directory.ts)).
|
|
13
|
+
- **Type safety** — use of `any` instead of `unknown`, missing type guards, incorrect use of types in [src/types/](src/types/).
|
|
14
|
+
- **Tests** — whether new behaviour is covered in [tests/](tests/); flag any missing cases across document, collection, or schema tests.
|
|
15
|
+
- **Public API** — unintended changes to [src/types/fylo.d.ts](src/types/fylo.d.ts) or [src/types/query.d.ts](src/types/query.d.ts).
|
|
16
|
+
- **CI** — whether the workflow files in [.github/workflows/](.github/workflows/) are still valid for the change.
|
|
17
|
+
4. Post the review as inline comments using `gh pr review <arg> --comment --body "<feedback>"`.
|
|
18
|
+
Group feedback by file. Prefix each point with **[suggestion]**, **[issue]**, or **[nit]**.
|
|
19
|
+
5. Summarise the overall verdict: Approve / Request changes / Comment only.
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: "Rebase the current branch onto the latest main from origin"
|
|
3
|
+
agent: "agent"
|
|
4
|
+
tools: [runInTerminal]
|
|
5
|
+
---
|
|
6
|
+
Safely bring the current branch up to date with the latest `main` from origin.
|
|
7
|
+
|
|
8
|
+
1. Confirm the current branch with `git branch --show-current`. If already on `main`, just run `git pull` and stop.
|
|
9
|
+
2. Stash any uncommitted changes with `git stash push -m "sync-main auto-stash"` and note whether anything was stashed.
|
|
10
|
+
3. Fetch the latest: `git fetch origin main`.
|
|
11
|
+
4. Rebase the current branch onto `origin/main`: `git rebase origin/main`.
|
|
12
|
+
5. If the rebase has conflicts, list the conflicting files and stop — ask the user to resolve them, then run `git rebase --continue`.
|
|
13
|
+
6. If a stash was created in step 2, restore it with `git stash pop`.
|
|
14
|
+
7. Report: commits rebased, files changed, any stash restored.
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
name: CI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [release/*]
|
|
6
|
+
pull_request:
|
|
7
|
+
branches: [main]
|
|
8
|
+
|
|
9
|
+
jobs:
|
|
10
|
+
test:
|
|
11
|
+
name: Test (Bun ${{ matrix.bun-version }})
|
|
12
|
+
runs-on: ubuntu-latest
|
|
13
|
+
|
|
14
|
+
strategy:
|
|
15
|
+
matrix:
|
|
16
|
+
bun-version: [latest, 1.2.x]
|
|
17
|
+
|
|
18
|
+
steps:
|
|
19
|
+
- name: Checkout
|
|
20
|
+
uses: actions/checkout@v4
|
|
21
|
+
|
|
22
|
+
- name: Setup Bun
|
|
23
|
+
uses: oven-sh/setup-bun@v2
|
|
24
|
+
with:
|
|
25
|
+
bun-version: ${{ matrix.bun-version }}
|
|
26
|
+
|
|
27
|
+
- name: Install dependencies
|
|
28
|
+
run: bun install --frozen-lockfile
|
|
29
|
+
|
|
30
|
+
- name: Type check
|
|
31
|
+
run: bun run typecheck
|
|
32
|
+
|
|
33
|
+
- name: Lint
|
|
34
|
+
run: bun run lint
|
|
35
|
+
|
|
36
|
+
- name: Run tests
|
|
37
|
+
run: bun test
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
name: Publish
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [release/*]
|
|
6
|
+
|
|
7
|
+
jobs:
|
|
8
|
+
test:
|
|
9
|
+
name: Test before publish
|
|
10
|
+
runs-on: ubuntu-latest
|
|
11
|
+
|
|
12
|
+
steps:
|
|
13
|
+
- name: Checkout
|
|
14
|
+
uses: actions/checkout@v4
|
|
15
|
+
|
|
16
|
+
- name: Setup Bun
|
|
17
|
+
uses: oven-sh/setup-bun@v2
|
|
18
|
+
with:
|
|
19
|
+
bun-version: latest
|
|
20
|
+
|
|
21
|
+
- name: Install dependencies
|
|
22
|
+
run: bun install --frozen-lockfile
|
|
23
|
+
|
|
24
|
+
- name: Type check
|
|
25
|
+
run: bun run typecheck
|
|
26
|
+
|
|
27
|
+
- name: Lint
|
|
28
|
+
run: bun run lint
|
|
29
|
+
|
|
30
|
+
- name: Run tests
|
|
31
|
+
run: bun test
|
|
32
|
+
|
|
33
|
+
publish:
|
|
34
|
+
name: Publish to npm
|
|
35
|
+
runs-on: ubuntu-latest
|
|
36
|
+
needs: test
|
|
37
|
+
permissions:
|
|
38
|
+
contents: write # create git tags
|
|
39
|
+
|
|
40
|
+
steps:
|
|
41
|
+
- name: Checkout
|
|
42
|
+
uses: actions/checkout@v4
|
|
43
|
+
|
|
44
|
+
- name: Setup Bun
|
|
45
|
+
uses: oven-sh/setup-bun@v2
|
|
46
|
+
with:
|
|
47
|
+
bun-version: latest
|
|
48
|
+
|
|
49
|
+
- name: Install dependencies
|
|
50
|
+
run: bun install --frozen-lockfile
|
|
51
|
+
|
|
52
|
+
- name: Setup Node and upgrade npm
|
|
53
|
+
uses: actions/setup-node@v4
|
|
54
|
+
with:
|
|
55
|
+
node-version: '20'
|
|
56
|
+
registry-url: 'https://registry.npmjs.org'
|
|
57
|
+
|
|
58
|
+
- name: Upgrade npm
|
|
59
|
+
run: npm install -g npm@latest
|
|
60
|
+
|
|
61
|
+
- name: Resolve version from branch
|
|
62
|
+
id: version
|
|
63
|
+
run: |
|
|
64
|
+
BRANCH="${GITHUB_REF#refs/heads/}"
|
|
65
|
+
VERSION="${BRANCH#release/}"
|
|
66
|
+
PKG_VERSION=$(bun -e "console.log(require('./package.json').version)")
|
|
67
|
+
if [ "$VERSION" != "$PKG_VERSION" ]; then
|
|
68
|
+
echo "Branch version ($VERSION) does not match package.json ($PKG_VERSION). Skipping publish."
|
|
69
|
+
exit 1
|
|
70
|
+
fi
|
|
71
|
+
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
|
|
72
|
+
|
|
73
|
+
- name: Verify NPM token
|
|
74
|
+
run: |
|
|
75
|
+
if [ -z "${NPM_TOKEN}" ]; then
|
|
76
|
+
echo "NPM_TOKEN secret is not configured."
|
|
77
|
+
exit 1
|
|
78
|
+
fi
|
|
79
|
+
env:
|
|
80
|
+
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
|
|
81
|
+
|
|
82
|
+
- name: Publish to npm
|
|
83
|
+
run: |
|
|
84
|
+
npm publish --access public
|
|
85
|
+
env:
|
|
86
|
+
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
|
|
87
|
+
|
|
88
|
+
- name: Create and push version tag
|
|
89
|
+
run: |
|
|
90
|
+
git config user.name "github-actions[bot]"
|
|
91
|
+
git config user.email "github-actions[bot]@users.noreply.github.com"
|
|
92
|
+
git tag -f -a "v${{ steps.version.outputs.version }}" -m "v${{ steps.version.outputs.version }}"
|
|
93
|
+
git push origin "v${{ steps.version.outputs.version }}" --force
|
|
94
|
+
|
|
95
|
+
- name: Create GitHub release
|
|
96
|
+
run: |
|
|
97
|
+
gh release create "v${{ steps.version.outputs.version }}" \
|
|
98
|
+
--title "v${{ steps.version.outputs.version }}" \
|
|
99
|
+
--generate-notes
|
|
100
|
+
env:
|
|
101
|
+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
package/.prettierrc
ADDED
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Vyckr
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,230 @@
|
|
|
1
|
+
# Fylo
|
|
2
|
+
|
|
3
|
+
S3-backed NoSQL document store with SQL parsing, Redis pub/sub for real-time events, and a CLI.
|
|
4
|
+
|
|
5
|
+
Documents are stored as **S3 key paths** — not file contents. Each document produces two keys per field: a **data key** (`{ttid}/{field}/{value}`) for full-doc retrieval and an **index key** (`{field}/{value}/{ttid}`) for query lookups. This enables fast reads and filtered queries without a traditional database engine.
|
|
6
|
+
|
|
7
|
+
Built for **serverless** runtimes (AWS Lambda, Cloudflare Workers) — no persistent in-memory state, lazy connections, minimal cold-start overhead.
|
|
8
|
+
|
|
9
|
+
## Install
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
bun add @vyckr/fylo
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## Environment Variables
|
|
16
|
+
|
|
17
|
+
| Variable | Purpose |
|
|
18
|
+
|----------|---------|
|
|
19
|
+
| `BUCKET_PREFIX` | S3 bucket name prefix |
|
|
20
|
+
| `S3_ACCESS_KEY_ID` / `AWS_ACCESS_KEY_ID` | S3 credentials |
|
|
21
|
+
| `S3_SECRET_ACCESS_KEY` / `AWS_SECRET_ACCESS_KEY` | S3 credentials |
|
|
22
|
+
| `S3_REGION` / `AWS_REGION` | S3 region |
|
|
23
|
+
| `S3_ENDPOINT` / `AWS_ENDPOINT` | S3 endpoint (for LocalStack, MinIO, etc.) |
|
|
24
|
+
| `REDIS_URL` | Redis connection URL (default: `redis://localhost:6379`) |
|
|
25
|
+
| `LOGGING` | Enable debug logging |
|
|
26
|
+
| `STRICT` | Enable schema validation via CHEX |
|
|
27
|
+
|
|
28
|
+
## Usage
|
|
29
|
+
|
|
30
|
+
### CRUD — NoSQL API
|
|
31
|
+
|
|
32
|
+
```typescript
|
|
33
|
+
import Fylo from "@vyckr/fylo"
|
|
34
|
+
|
|
35
|
+
const fylo = new Fylo()
|
|
36
|
+
|
|
37
|
+
// Collections
|
|
38
|
+
await Fylo.createCollection("users")
|
|
39
|
+
|
|
40
|
+
// Create
|
|
41
|
+
const _id = await fylo.putData<_user>("users", { name: "John Doe", age: 30 })
|
|
42
|
+
|
|
43
|
+
// Read one
|
|
44
|
+
const user = await Fylo.getDoc<_user>("users", _id).once()
|
|
45
|
+
|
|
46
|
+
// Read many
|
|
47
|
+
for await (const doc of Fylo.findDocs<_user>("users", { $limit: 10 }).collect()) {
|
|
48
|
+
console.log(doc)
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
// Update one
|
|
52
|
+
await fylo.patchDoc<_user>("users", { [_id]: { age: 31 } })
|
|
53
|
+
|
|
54
|
+
// Update many
|
|
55
|
+
const updated = await fylo.patchDocs<_user>("users", {
|
|
56
|
+
$where: { $ops: [{ age: { $gte: 30 } }] },
|
|
57
|
+
$set: { age: 31 }
|
|
58
|
+
})
|
|
59
|
+
|
|
60
|
+
// Delete one
|
|
61
|
+
await fylo.delDoc("users", _id)
|
|
62
|
+
|
|
63
|
+
// Delete many
|
|
64
|
+
const deleted = await fylo.delDocs<_user>("users", {
|
|
65
|
+
$ops: [{ name: { $like: "%Doe%" } }]
|
|
66
|
+
})
|
|
67
|
+
|
|
68
|
+
// Drop
|
|
69
|
+
await Fylo.dropCollection("users")
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
### CRUD — SQL API
|
|
73
|
+
|
|
74
|
+
```typescript
|
|
75
|
+
const fylo = new Fylo()
|
|
76
|
+
|
|
77
|
+
await fylo.executeSQL(`CREATE TABLE users`)
|
|
78
|
+
|
|
79
|
+
const _id = await fylo.executeSQL<_user>(`INSERT INTO users (name, age) VALUES ('John Doe', 30)`)
|
|
80
|
+
|
|
81
|
+
const docs = await fylo.executeSQL<_user>(`SELECT * FROM users LIMIT 10`)
|
|
82
|
+
|
|
83
|
+
await fylo.executeSQL<_user>(`UPDATE users SET age = 31 WHERE name = 'John Doe'`)
|
|
84
|
+
|
|
85
|
+
await fylo.executeSQL<_user>(`DELETE FROM users WHERE name LIKE '%Doe%'`)
|
|
86
|
+
|
|
87
|
+
await fylo.executeSQL(`DROP TABLE users`)
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
### Query Operators
|
|
91
|
+
|
|
92
|
+
```typescript
|
|
93
|
+
// Equality
|
|
94
|
+
{ $ops: [{ status: { $eq: "active" } }] }
|
|
95
|
+
|
|
96
|
+
// Not equal
|
|
97
|
+
{ $ops: [{ status: { $ne: "archived" } }] }
|
|
98
|
+
|
|
99
|
+
// Numeric range
|
|
100
|
+
{ $ops: [{ age: { $gte: 18, $lt: 65 } }] }
|
|
101
|
+
|
|
102
|
+
// Pattern matching
|
|
103
|
+
{ $ops: [{ email: { $like: "%@gmail.com" } }] }
|
|
104
|
+
|
|
105
|
+
// Array contains
|
|
106
|
+
{ $ops: [{ tags: { $contains: "urgent" } }] }
|
|
107
|
+
|
|
108
|
+
// Multiple ops use OR semantics — matches if any op is satisfied
|
|
109
|
+
{ $ops: [
|
|
110
|
+
{ status: { $eq: "active" } },
|
|
111
|
+
{ priority: { $gte: 5 } }
|
|
112
|
+
]}
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
### Joins
|
|
116
|
+
|
|
117
|
+
```typescript
|
|
118
|
+
const results = await Fylo.joinDocs<_post, _user>({
|
|
119
|
+
$leftCollection: "posts",
|
|
120
|
+
$rightCollection: "users",
|
|
121
|
+
$mode: "inner", // "inner" | "left" | "right" | "outer"
|
|
122
|
+
$on: { userId: { $eq: "id" } },
|
|
123
|
+
$select: ["title", "name"],
|
|
124
|
+
$limit: 50
|
|
125
|
+
})
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
### Real-Time Streaming
|
|
129
|
+
|
|
130
|
+
```typescript
|
|
131
|
+
// Stream new/updated documents
|
|
132
|
+
for await (const doc of Fylo.findDocs<_user>("users")) {
|
|
133
|
+
console.log(doc)
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
// Stream deletions
|
|
137
|
+
for await (const _id of Fylo.findDocs<_user>("users").onDelete()) {
|
|
138
|
+
console.log("deleted:", _id)
|
|
139
|
+
}
|
|
140
|
+
|
|
141
|
+
// Watch a single document
|
|
142
|
+
for await (const doc of Fylo.getDoc<_user>("users", _id)) {
|
|
143
|
+
console.log(doc)
|
|
144
|
+
}
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
### Bulk Import / Export
|
|
148
|
+
|
|
149
|
+
```typescript
|
|
150
|
+
const fylo = new Fylo()
|
|
151
|
+
|
|
152
|
+
// Import from JSON array or NDJSON URL
|
|
153
|
+
const count = await fylo.importBulkData<_user>("users", new URL("https://example.com/users.json"), 1000)
|
|
154
|
+
|
|
155
|
+
// Export all documents
|
|
156
|
+
for await (const doc of Fylo.exportBulkData<_user>("users")) {
|
|
157
|
+
console.log(doc)
|
|
158
|
+
}
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
### Rollback
|
|
162
|
+
|
|
163
|
+
Every write is tracked as a transaction. If a batch write partially fails, Fylo automatically rolls back. You can also trigger it manually:
|
|
164
|
+
|
|
165
|
+
```typescript
|
|
166
|
+
const fylo = new Fylo()
|
|
167
|
+
await fylo.putData("users", { name: "test" })
|
|
168
|
+
await fylo.rollback() // undoes all writes in this instance
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
### CLI
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
fylo.query "SELECT * FROM users WHERE age > 25 LIMIT 10"
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
### Schema Validation
|
|
178
|
+
|
|
179
|
+
When `STRICT` is set, documents are validated against CHEX schemas before writes:
|
|
180
|
+
|
|
181
|
+
```bash
|
|
182
|
+
STRICT=true bun run start
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
Schemas are `.d.ts` interface declarations generated by [`@vyckr/chex`](https://github.com/vyckr/chex).
|
|
186
|
+
|
|
187
|
+
## Development
|
|
188
|
+
|
|
189
|
+
```bash
|
|
190
|
+
bun test # Run all tests
|
|
191
|
+
bun run build # Compile TypeScript
|
|
192
|
+
bun run typecheck # Type-check without emitting
|
|
193
|
+
bun run lint # ESLint
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
### Local S3 (LocalStack)
|
|
197
|
+
|
|
198
|
+
```bash
|
|
199
|
+
docker compose up aws
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
This starts LocalStack on `localhost:4566`. Set `S3_ENDPOINT=http://localhost:4566` to route S3 calls locally.
|
|
203
|
+
|
|
204
|
+
## Security
|
|
205
|
+
|
|
206
|
+
### What Fylo does NOT provide
|
|
207
|
+
|
|
208
|
+
Fylo is a low-level storage abstraction. The following must be implemented by the integrating application:
|
|
209
|
+
|
|
210
|
+
- **Authentication** — Fylo has no concept of users or sessions. Any caller with access to the Fylo instance can read and write any collection.
|
|
211
|
+
- **Authorization** — `executeSQL` and all document operations accept any collection name with no permission check. In multi-tenant applications, a caller can access any collection unless the integrator enforces a boundary above Fylo.
|
|
212
|
+
- **Rate limiting** — There is no built-in request throttling. An attacker with access to the instance can flood S3 with requests or trigger expensive operations without restriction. Add rate limiting and document-size limits in your service layer.
|
|
213
|
+
|
|
214
|
+
### Secure configuration
|
|
215
|
+
|
|
216
|
+
| Concern | Guidance |
|
|
217
|
+
|---------|----------|
|
|
218
|
+
| AWS credentials | Never commit credentials to version control. Use IAM instance roles or inject via CI secrets. Rotate any credentials that have been exposed. |
|
|
219
|
+
| `ENCRYPTION_KEY` | Must be at least 32 characters. Use a high-entropy random value. |
|
|
220
|
+
| `CIPHER_SALT` | Set a unique random value per deployment to prevent cross-instance precomputation attacks. |
|
|
221
|
+
| `REDIS_URL` | Always set explicitly. Use `rediss://` (TLS) in production with authentication credentials in the URL. |
|
|
222
|
+
| Collection names | Must match `^[a-z0-9][a-z0-9\-]*[a-z0-9]$`. Names are validated before any shell or S3 operation. |
|
|
223
|
+
|
|
224
|
+
### Encrypted fields
|
|
225
|
+
|
|
226
|
+
Fields listed in `$encrypted` in a collection schema are encrypted with AES-256-CBC. By default a random IV is used per write (non-deterministic). Pass `deterministic: true` to `Cipher.encrypt()` only for fields that require `$eq`/`$ne` queries — deterministic encryption leaks value equality to observers of stored ciphertext.
|
|
227
|
+
|
|
228
|
+
## License
|
|
229
|
+
|
|
230
|
+
MIT
|