@delma/fylo 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (60) hide show
  1. package/.env.example +16 -0
  2. package/.github/copilot-instructions.md +113 -0
  3. package/.github/prompts/issue.prompt.md +19 -0
  4. package/.github/prompts/pr.prompt.md +18 -0
  5. package/.github/prompts/release.prompt.md +49 -0
  6. package/.github/prompts/review-pr.prompt.md +19 -0
  7. package/.github/prompts/sync-main.prompt.md +14 -0
  8. package/.github/workflows/ci.yml +37 -0
  9. package/.github/workflows/publish.yml +101 -0
  10. package/.prettierrc +7 -0
  11. package/LICENSE +21 -0
  12. package/README.md +230 -0
  13. package/eslint.config.js +28 -0
  14. package/package.json +51 -0
  15. package/src/CLI +37 -0
  16. package/src/adapters/cipher.ts +174 -0
  17. package/src/adapters/redis.ts +71 -0
  18. package/src/adapters/s3.ts +67 -0
  19. package/src/core/directory.ts +418 -0
  20. package/src/core/extensions.ts +19 -0
  21. package/src/core/format.ts +486 -0
  22. package/src/core/parser.ts +876 -0
  23. package/src/core/query.ts +48 -0
  24. package/src/core/walker.ts +167 -0
  25. package/src/index.ts +1088 -0
  26. package/src/types/fylo.d.ts +139 -0
  27. package/src/types/index.d.ts +3 -0
  28. package/src/types/query.d.ts +73 -0
  29. package/tests/collection/truncate.test.ts +56 -0
  30. package/tests/data.ts +110 -0
  31. package/tests/index.ts +19 -0
  32. package/tests/integration/create.test.ts +57 -0
  33. package/tests/integration/delete.test.ts +147 -0
  34. package/tests/integration/edge-cases.test.ts +232 -0
  35. package/tests/integration/encryption.test.ts +176 -0
  36. package/tests/integration/export.test.ts +61 -0
  37. package/tests/integration/join-modes.test.ts +221 -0
  38. package/tests/integration/nested.test.ts +212 -0
  39. package/tests/integration/operators.test.ts +167 -0
  40. package/tests/integration/read.test.ts +203 -0
  41. package/tests/integration/rollback.test.ts +105 -0
  42. package/tests/integration/update.test.ts +130 -0
  43. package/tests/mocks/cipher.ts +55 -0
  44. package/tests/mocks/redis.ts +13 -0
  45. package/tests/mocks/s3.ts +114 -0
  46. package/tests/schemas/album.d.ts +5 -0
  47. package/tests/schemas/album.json +5 -0
  48. package/tests/schemas/comment.d.ts +7 -0
  49. package/tests/schemas/comment.json +7 -0
  50. package/tests/schemas/photo.d.ts +7 -0
  51. package/tests/schemas/photo.json +7 -0
  52. package/tests/schemas/post.d.ts +6 -0
  53. package/tests/schemas/post.json +6 -0
  54. package/tests/schemas/tip.d.ts +7 -0
  55. package/tests/schemas/tip.json +7 -0
  56. package/tests/schemas/todo.d.ts +6 -0
  57. package/tests/schemas/todo.json +6 -0
  58. package/tests/schemas/user.d.ts +23 -0
  59. package/tests/schemas/user.json +23 -0
  60. package/tsconfig.json +19 -0
package/.env.example ADDED
@@ -0,0 +1,16 @@
1
+ LOGGING=
2
+ STRICT=
3
+ S3_REGION=ca-central-1
4
+ S3_ACCESS_KEY_ID=HELLO
5
+ S3_SECRET_ACCESS_KEY=WORLD
6
+ S3_ENDPOINT=https//example.com
7
+ BUCKET_PREFIX="byos-test"
8
+ SCHEMA_DIR=/path/to/schema/dir
9
+ REDIS_URL=redis://localhost:6379
10
+ REDIS_CONN_TIMEOUT=5000
11
+ REDIS_IDLE_TIMEOUT=30000
12
+ REDIS_AUTO_RECONNECT=
13
+ REDIS_MAX_RETRIES=10
14
+ REDIS_ENABLE_OFFLINE_QUEUE=
15
+ REDIS_ENABLE_AUTO_PIPELINING=
16
+ REDIS_TLS=
@@ -0,0 +1,113 @@
1
+ # FYLO — Project Guidelines
2
+
3
+ ## Overview
4
+
5
+ FYLO (`@vyckr/fylo`) is an S3-backed NoSQL document store with SQL parsing, Redis pub/sub for real-time events, and a CLI. Documents are stored as S3 key paths — not as file contents — with dual key layouts for data access and indexed queries.
6
+
7
+ **Assume a serverless deployment model** (e.g., AWS Lambda, Cloudflare Workers). This means:
8
+ - No persistent in-memory state across invocations — every request starts cold
9
+ - Distributed coordination (e.g., TTID uniqueness) must use external stores like Redis, not in-process caches
10
+ - Avoid long-lived connections, background threads, or singleton patterns that assume process longevity
11
+ - Keep cold-start overhead minimal — lazy initialization over eager setup
12
+
13
+ ## Architecture
14
+
15
+ ### Key Storage Format
16
+
17
+ - **Data keys**: `{ttid}/{field}/{value}` — keyed by document ID for full-doc retrieval
18
+ - **Index keys**: `{field}/{value}/{ttid}` — keyed by field for query lookups
19
+ - Nested objects flatten to path segments: `address/city/Toronto`
20
+ - Forward slashes in values are escaped with an ASCII substitute
21
+
22
+ ### Core Modules
23
+
24
+ | Module | Responsibility |
25
+ |--------|---------------|
26
+ | `src/index.ts` | Main `Fylo` class — CRUD, SQL execution, joins, bulk ops |
27
+ | `src/core/parser.ts` | SQL lexer/parser — tokenizes SQL into query objects |
28
+ | `src/core/query.ts` | Converts `$ops` into glob patterns for S3 key matching |
29
+ | `src/core/walker.ts` | S3 key traversal, document data retrieval, Redis event streaming |
30
+ | `src/core/directory.ts` | Key extraction, reconstruction, rollback tracking |
31
+ | `src/core/format.ts` | Console formatting for query output |
32
+ | `src/adapters/s3.ts` | S3 adapter (Bun S3Client) |
33
+ | `src/adapters/redis.ts` | Redis adapter (Bun RedisClient) |
34
+ | `src/cli/index.ts` | CLI entry point (`fylo.query`) |
35
+
36
+ ### Folder Structure
37
+
38
+ ```
39
+ src/
40
+ index.ts # Public API — main Fylo class
41
+ adapters/ # I/O boundary abstractions (S3, Redis)
42
+ core/ # Internal domain logic (parser, query, walker, directory)
43
+ cli/ # CLI entry point
44
+ types/ # Type declarations (.d.ts only — separate from implementation)
45
+ tests/
46
+ data.ts # Shared test data URLs
47
+ index.ts # Test barrel
48
+ mocks/ # Mock adapters (S3, Redis) for testing
49
+ schemas/ # CHEX-generated test schemas (.d.ts + .json)
50
+ integration/ # End-to-end tests (CRUD, operators, joins, edge cases)
51
+ ```
52
+
53
+ ### Dependencies
54
+
55
+ - **`@vyckr/ttid`** — Time-based unique ID system. `TTID.generate()` creates new IDs; `TTID.generate(existingId)` creates a versioned ID sharing the same creation-time prefix.
56
+ - **`@vyckr/chex`** — Schema validation. Generates `interface` declarations in `.d.ts` files. Generic constraints must use `Record<string, any>` (not `Record<string, unknown>`) to accept these interfaces.
57
+ - **`Bun.Glob`** — Pattern matching for queries. Does NOT support negation extglob `!(pattern)`. Operators like `$ne`, `$gt`, `$lt` use broad globs with post-filtering instead.
58
+
59
+ ## Engineering Standards
60
+
61
+ - **SOLID principles**: Single responsibility per class/method, depend on abstractions (e.g., S3/Redis adapters), open for extension without modifying core logic
62
+ - **Clean code**: Descriptive naming, small focused functions, no dead code or commented-out blocks, DRY without premature abstraction
63
+ - **Test discipline**: When changing `src/` code, update or add corresponding tests in `tests/` — never leave tests stale after a behaviour change
64
+ - **Error handling**: Fail fast with meaningful errors at system boundaries; use rollback mechanisms for partial writes
65
+ - **No magic values**: Use constants or environment variables; avoid hardcoded strings/numbers in logic
66
+ - **Type safety**: Leverage TypeScript's type system fully — avoid `any` in implementation code, prefer narrow types, and validate at I/O boundaries
67
+
68
+ ## Code Style
69
+
70
+ - **Runtime**: Bun (ESNext target, ES modules)
71
+ - **Strict TypeScript**: `strict: true`, `noImplicitReturns`, `isolatedModules`
72
+ - **ESLint** enforces `@typescript-eslint/no-explicit-any` in `src/` and `tests/` — use it only in type declarations (`.d.ts`)
73
+ - **No default exports** except the main `Fylo` class
74
+ - Prefer `class` with `static` methods for modules (no standalone functions)
75
+ - Use `_ttid` branded type for document IDs — never plain `string`
76
+ - Prefix internal/test type names with underscore: `_post`, `_album`, `_storeQuery`
77
+ - Type declarations live in `src/types/*.d.ts` — keep separate from implementation
78
+
79
+ ## Build & Test
80
+
81
+ ```bash
82
+ bun test # Run all tests
83
+ bun run build # Compile TypeScript
84
+ bun run typecheck # Type-check without emitting
85
+ bun run lint # ESLint
86
+ ```
87
+
88
+ - Tests use `bun:test` — `describe`, `test`, `expect`, `mock`, `beforeAll`, `afterAll`
89
+ - S3 and Redis are mocked via `mock.module()` in every test file using `tests/mocks/s3.ts` and `tests/mocks/redis.ts`
90
+ - Test schemas live in `tests/schemas/*.d.ts` as global `interface` declarations (generated by CHEX)
91
+ - Test data URLs are centralized in `tests/data.ts`
92
+
93
+ ## Conventions
94
+
95
+ - Collection names may contain hyphens (e.g., `ec-test`, `jm-album`) — the parser supports this
96
+ - Nested field access in SQL uses dot notation (`address.city`) which the parser converts to slash-separated paths (`address/city`)
97
+ - `putData` creates documents; `patchDoc` updates them (deletes old keys, writes new ones)
98
+ - `getDocData` retrieves keys for a specific TTID — filters by exact ID, not just prefix
99
+ - Query `$ops` use OR semantics — a document matches if it satisfies at least one operator
100
+ - `$limit` on queries without `$ops` uses S3 `maxKeys`; with `$ops` it post-filters after glob matching
101
+
102
+ ## Environment Variables
103
+
104
+ | Variable | Purpose |
105
+ |----------|---------|
106
+ | `BUCKET_PREFIX` | S3 bucket name prefix |
107
+ | `S3_ACCESS_KEY_ID` / `AWS_ACCESS_KEY_ID` | S3 credentials |
108
+ | `S3_SECRET_ACCESS_KEY` / `AWS_SECRET_ACCESS_KEY` | S3 credentials |
109
+ | `S3_REGION` / `AWS_REGION` | S3 region |
110
+ | `S3_ENDPOINT` / `AWS_ENDPOINT` | S3 endpoint (for compatible stores) |
111
+ | `REDIS_URL` | Redis connection URL |
112
+ | `LOGGING` | Enable debug logging |
113
+ | `STRICT` | Enable schema validation via CHEX |
@@ -0,0 +1,19 @@
1
+ ---
2
+ description: "Create a GitHub issue for a bug or feature request"
3
+ argument-hint: "Describe the bug or feature request"
4
+ agent: "agent"
5
+ tools: [runInTerminal]
6
+ ---
7
+ Create a GitHub issue. The user's description is provided as the argument.
8
+
9
+ Determine whether the description sounds like a bug or a feature request, then create the issue with `gh issue create` using the following structure:
10
+
11
+ **For a bug:**
12
+ - Title: "fix: <short description>"
13
+ - Body sections: **Describe the bug**, **Steps to reproduce**, **Expected behaviour**, **Actual behaviour**, **Possible cause** (reference relevant lines in `src/index.ts` if applicable), **Environment** (Bun version, TypeScript version, AWS/S3 endpoint details).
14
+
15
+ **For a feature request:**
16
+ - Title: "feat: <short description>"
17
+ - Body sections: **Problem**, **Proposed solution**, **Alternatives considered**, **Affected API** (list any methods in `src/types/fylo.d.ts` or `src/types/query.d.ts` that would change).
18
+
19
+ Apply the appropriate label (`bug` or `enhancement`) via `--label`. Print the issue URL when done.
@@ -0,0 +1,18 @@
1
+ ---
2
+ description: "Create a pull request for the current branch into main"
3
+ agent: "agent"
4
+ tools: [runInTerminal]
5
+ ---
6
+ Create a pull request for the current branch into `main`.
7
+
8
+ 1. Run `git status` and stop if there are uncommitted changes — ask the user to commit or stash them first.
9
+ 2. Run `git log main..HEAD --oneline` to list commits on this branch.
10
+ 3. Run `git diff main...HEAD` to understand all changes.
11
+ 4. Push the branch to origin if it has no upstream: `git push -u origin HEAD`.
12
+ 5. Create the PR with `gh pr create` using:
13
+ - A concise title (≤ 70 chars) derived from the branch name and commits.
14
+ - A body with three sections:
15
+ - **Summary** — bullet list of what changed and why.
16
+ - **Test plan** — checklist of how to verify the changes (reference `bun test` where relevant).
17
+ - **Breaking changes** — any changes to public API in [src/types/fylo.d.ts](src/types/fylo.d.ts) or [src/types/query.d.ts](src/types/query.d.ts); write "None" if there are none.
18
+ 6. Print the PR URL.
@@ -0,0 +1,49 @@
1
+ ---
2
+ description: "Create a release branch, publish to npm via CI, then merge to main"
3
+ agent: "agent"
4
+ tools: [runInTerminal]
5
+ ---
6
+ Create a release branch, publish to npm via CI, then merge to main.
7
+
8
+ 1. Run `bun test` and stop if any tests fail.
9
+
10
+ 2. Determine the new version automatically based on unreleased commits:
11
+ `git log $(git describe --tags --abbrev=0 2>/dev/null || git rev-list --max-parents=0 HEAD)..HEAD --oneline`
12
+
13
+ Apply these rules to select the bump type:
14
+ - **major** — any commit with a `!` breaking-change marker (e.g. `feat!:`, `fix!:`) or a `BREAKING CHANGE` footer.
15
+ - **minor** — one or more `feat:` commits and no breaking changes.
16
+ - **patch** — only `fix:`, `chore:`, `docs:`, `refactor:`, `test:`, or `perf:` commits.
17
+
18
+ Compute the new version by incrementing the corresponding part of the current `"version"` in [package.json](package.json) and resetting lower parts to zero. Show the chosen version and the reasoning to the user before proceeding.
19
+
20
+ 3. Update `"version"` in [package.json](package.json) to the new version.
21
+
22
+ 4. Fetch the latest main and create a release branch from it:
23
+ ```
24
+ git fetch origin main
25
+ git checkout -b release/<version> origin/main
26
+ ```
27
+
28
+ 5. Stage all changes and commit:
29
+ `git add -A && git commit -m "chore: release v<version>"`
30
+
31
+ 6. Push the branch:
32
+ `git push -u origin release/<version>`
33
+
34
+ 7. Tell the user that the `publish` workflow will now run on GitHub Actions:
35
+ - It verifies the branch name matches `package.json` version.
36
+ - It runs tests, publishes to npm, creates a git tag, and opens a GitHub release.
37
+ - The NPM_TOKEN secret must be set in repo Settings → Secrets → Actions.
38
+
39
+ 8. Once the workflow passes (user confirms), create a PR and merge it to main:
40
+ ```
41
+ gh pr create --title "chore: release v<version>" --body "Release v<version>" --base main --head release/<version>
42
+ gh pr merge --merge --delete-branch
43
+ ```
44
+
45
+ 9. Switch back to main and pull:
46
+ ```
47
+ git checkout main
48
+ git pull
49
+ ```
@@ -0,0 +1,19 @@
1
+ ---
2
+ description: "Review a pull request by number or URL"
3
+ argument-hint: "PR number or URL (e.g. 42)"
4
+ agent: "agent"
5
+ tools: [runInTerminal]
6
+ ---
7
+ Review the pull request given as an argument (PR number or URL).
8
+
9
+ 1. Fetch the PR details: `gh pr view <arg> --json title,body,headRefName,baseRefName,files`
10
+ 2. Fetch the diff: `gh pr diff <arg>`
11
+ 3. Review the changes with focus on:
12
+ - **Correctness** — logic errors, edge cases missed in query parsing ([src/core/parser.ts](src/core/parser.ts)), S3 operations ([src/adapters/s3.ts](src/adapters/s3.ts)), or directory/index management ([src/core/directory.ts](src/core/directory.ts)).
13
+ - **Type safety** — use of `any` instead of `unknown`, missing type guards, incorrect use of types in [src/types/](src/types/).
14
+ - **Tests** — whether new behaviour is covered in [tests/](tests/); flag any missing cases across document, collection, or schema tests.
15
+ - **Public API** — unintended changes to [src/types/fylo.d.ts](src/types/fylo.d.ts) or [src/types/query.d.ts](src/types/query.d.ts).
16
+ - **CI** — whether the workflow files in [.github/workflows/](.github/workflows/) are still valid for the change.
17
+ 4. Post the review as inline comments using `gh pr review <arg> --comment --body "<feedback>"`.
18
+ Group feedback by file. Prefix each point with **[suggestion]**, **[issue]**, or **[nit]**.
19
+ 5. Summarise the overall verdict: Approve / Request changes / Comment only.
@@ -0,0 +1,14 @@
1
+ ---
2
+ description: "Rebase the current branch onto the latest main from origin"
3
+ agent: "agent"
4
+ tools: [runInTerminal]
5
+ ---
6
+ Safely bring the current branch up to date with the latest `main` from origin.
7
+
8
+ 1. Confirm the current branch with `git branch --show-current`. If already on `main`, just run `git pull` and stop.
9
+ 2. Stash any uncommitted changes with `git stash push -m "sync-main auto-stash"` and note whether anything was stashed.
10
+ 3. Fetch the latest: `git fetch origin main`.
11
+ 4. Rebase the current branch onto `origin/main`: `git rebase origin/main`.
12
+ 5. If the rebase has conflicts, list the conflicting files and stop — ask the user to resolve them, then run `git rebase --continue`.
13
+ 6. If a stash was created in step 2, restore it with `git stash pop`.
14
+ 7. Report: commits rebased, files changed, any stash restored.
@@ -0,0 +1,37 @@
1
+ name: CI
2
+
3
+ on:
4
+ push:
5
+ branches: [release/*]
6
+ pull_request:
7
+ branches: [main]
8
+
9
+ jobs:
10
+ test:
11
+ name: Test (Bun ${{ matrix.bun-version }})
12
+ runs-on: ubuntu-latest
13
+
14
+ strategy:
15
+ matrix:
16
+ bun-version: [latest, 1.2.x]
17
+
18
+ steps:
19
+ - name: Checkout
20
+ uses: actions/checkout@v4
21
+
22
+ - name: Setup Bun
23
+ uses: oven-sh/setup-bun@v2
24
+ with:
25
+ bun-version: ${{ matrix.bun-version }}
26
+
27
+ - name: Install dependencies
28
+ run: bun install --frozen-lockfile
29
+
30
+ - name: Type check
31
+ run: bun run typecheck
32
+
33
+ - name: Lint
34
+ run: bun run lint
35
+
36
+ - name: Run tests
37
+ run: bun test
@@ -0,0 +1,101 @@
1
+ name: Publish
2
+
3
+ on:
4
+ push:
5
+ branches: [release/*]
6
+
7
+ jobs:
8
+ test:
9
+ name: Test before publish
10
+ runs-on: ubuntu-latest
11
+
12
+ steps:
13
+ - name: Checkout
14
+ uses: actions/checkout@v4
15
+
16
+ - name: Setup Bun
17
+ uses: oven-sh/setup-bun@v2
18
+ with:
19
+ bun-version: latest
20
+
21
+ - name: Install dependencies
22
+ run: bun install --frozen-lockfile
23
+
24
+ - name: Type check
25
+ run: bun run typecheck
26
+
27
+ - name: Lint
28
+ run: bun run lint
29
+
30
+ - name: Run tests
31
+ run: bun test
32
+
33
+ publish:
34
+ name: Publish to npm
35
+ runs-on: ubuntu-latest
36
+ needs: test
37
+ permissions:
38
+ contents: write # create git tags
39
+
40
+ steps:
41
+ - name: Checkout
42
+ uses: actions/checkout@v4
43
+
44
+ - name: Setup Bun
45
+ uses: oven-sh/setup-bun@v2
46
+ with:
47
+ bun-version: latest
48
+
49
+ - name: Install dependencies
50
+ run: bun install --frozen-lockfile
51
+
52
+ - name: Setup Node and upgrade npm
53
+ uses: actions/setup-node@v4
54
+ with:
55
+ node-version: '20'
56
+ registry-url: 'https://registry.npmjs.org'
57
+
58
+ - name: Upgrade npm
59
+ run: npm install -g npm@latest
60
+
61
+ - name: Resolve version from branch
62
+ id: version
63
+ run: |
64
+ BRANCH="${GITHUB_REF#refs/heads/}"
65
+ VERSION="${BRANCH#release/}"
66
+ PKG_VERSION=$(bun -e "console.log(require('./package.json').version)")
67
+ if [ "$VERSION" != "$PKG_VERSION" ]; then
68
+ echo "Branch version ($VERSION) does not match package.json ($PKG_VERSION). Skipping publish."
69
+ exit 1
70
+ fi
71
+ echo "version=$VERSION" >> "$GITHUB_OUTPUT"
72
+
73
+ - name: Verify NPM token
74
+ run: |
75
+ if [ -z "${NPM_TOKEN}" ]; then
76
+ echo "NPM_TOKEN secret is not configured."
77
+ exit 1
78
+ fi
79
+ env:
80
+ NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
81
+
82
+ - name: Publish to npm
83
+ run: |
84
+ npm publish --access public
85
+ env:
86
+ NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
87
+
88
+ - name: Create and push version tag
89
+ run: |
90
+ git config user.name "github-actions[bot]"
91
+ git config user.email "github-actions[bot]@users.noreply.github.com"
92
+ git tag -f -a "v${{ steps.version.outputs.version }}" -m "v${{ steps.version.outputs.version }}"
93
+ git push origin "v${{ steps.version.outputs.version }}" --force
94
+
95
+ - name: Create GitHub release
96
+ run: |
97
+ gh release create "v${{ steps.version.outputs.version }}" \
98
+ --title "v${{ steps.version.outputs.version }}" \
99
+ --generate-notes
100
+ env:
101
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
package/.prettierrc ADDED
@@ -0,0 +1,7 @@
1
+ {
2
+ "semi": false,
3
+ "singleQuote": true,
4
+ "tabWidth": 4,
5
+ "trailingComma": "none",
6
+ "printWidth": 100
7
+ }
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Vyckr
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,230 @@
1
+ # Fylo
2
+
3
+ S3-backed NoSQL document store with SQL parsing, Redis pub/sub for real-time events, and a CLI.
4
+
5
+ Documents are stored as **S3 key paths** — not file contents. Each document produces two keys per field: a **data key** (`{ttid}/{field}/{value}`) for full-doc retrieval and an **index key** (`{field}/{value}/{ttid}`) for query lookups. This enables fast reads and filtered queries without a traditional database engine.
6
+
7
+ Built for **serverless** runtimes (AWS Lambda, Cloudflare Workers) — no persistent in-memory state, lazy connections, minimal cold-start overhead.
8
+
9
+ ## Install
10
+
11
+ ```bash
12
+ bun add @vyckr/fylo
13
+ ```
14
+
15
+ ## Environment Variables
16
+
17
+ | Variable | Purpose |
18
+ |----------|---------|
19
+ | `BUCKET_PREFIX` | S3 bucket name prefix |
20
+ | `S3_ACCESS_KEY_ID` / `AWS_ACCESS_KEY_ID` | S3 credentials |
21
+ | `S3_SECRET_ACCESS_KEY` / `AWS_SECRET_ACCESS_KEY` | S3 credentials |
22
+ | `S3_REGION` / `AWS_REGION` | S3 region |
23
+ | `S3_ENDPOINT` / `AWS_ENDPOINT` | S3 endpoint (for LocalStack, MinIO, etc.) |
24
+ | `REDIS_URL` | Redis connection URL (default: `redis://localhost:6379`) |
25
+ | `LOGGING` | Enable debug logging |
26
+ | `STRICT` | Enable schema validation via CHEX |
27
+
28
+ ## Usage
29
+
30
+ ### CRUD — NoSQL API
31
+
32
+ ```typescript
33
+ import Fylo from "@vyckr/fylo"
34
+
35
+ const fylo = new Fylo()
36
+
37
+ // Collections
38
+ await Fylo.createCollection("users")
39
+
40
+ // Create
41
+ const _id = await fylo.putData<_user>("users", { name: "John Doe", age: 30 })
42
+
43
+ // Read one
44
+ const user = await Fylo.getDoc<_user>("users", _id).once()
45
+
46
+ // Read many
47
+ for await (const doc of Fylo.findDocs<_user>("users", { $limit: 10 }).collect()) {
48
+ console.log(doc)
49
+ }
50
+
51
+ // Update one
52
+ await fylo.patchDoc<_user>("users", { [_id]: { age: 31 } })
53
+
54
+ // Update many
55
+ const updated = await fylo.patchDocs<_user>("users", {
56
+ $where: { $ops: [{ age: { $gte: 30 } }] },
57
+ $set: { age: 31 }
58
+ })
59
+
60
+ // Delete one
61
+ await fylo.delDoc("users", _id)
62
+
63
+ // Delete many
64
+ const deleted = await fylo.delDocs<_user>("users", {
65
+ $ops: [{ name: { $like: "%Doe%" } }]
66
+ })
67
+
68
+ // Drop
69
+ await Fylo.dropCollection("users")
70
+ ```
71
+
72
+ ### CRUD — SQL API
73
+
74
+ ```typescript
75
+ const fylo = new Fylo()
76
+
77
+ await fylo.executeSQL(`CREATE TABLE users`)
78
+
79
+ const _id = await fylo.executeSQL<_user>(`INSERT INTO users (name, age) VALUES ('John Doe', 30)`)
80
+
81
+ const docs = await fylo.executeSQL<_user>(`SELECT * FROM users LIMIT 10`)
82
+
83
+ await fylo.executeSQL<_user>(`UPDATE users SET age = 31 WHERE name = 'John Doe'`)
84
+
85
+ await fylo.executeSQL<_user>(`DELETE FROM users WHERE name LIKE '%Doe%'`)
86
+
87
+ await fylo.executeSQL(`DROP TABLE users`)
88
+ ```
89
+
90
+ ### Query Operators
91
+
92
+ ```typescript
93
+ // Equality
94
+ { $ops: [{ status: { $eq: "active" } }] }
95
+
96
+ // Not equal
97
+ { $ops: [{ status: { $ne: "archived" } }] }
98
+
99
+ // Numeric range
100
+ { $ops: [{ age: { $gte: 18, $lt: 65 } }] }
101
+
102
+ // Pattern matching
103
+ { $ops: [{ email: { $like: "%@gmail.com" } }] }
104
+
105
+ // Array contains
106
+ { $ops: [{ tags: { $contains: "urgent" } }] }
107
+
108
+ // Multiple ops use OR semantics — matches if any op is satisfied
109
+ { $ops: [
110
+ { status: { $eq: "active" } },
111
+ { priority: { $gte: 5 } }
112
+ ]}
113
+ ```
114
+
115
+ ### Joins
116
+
117
+ ```typescript
118
+ const results = await Fylo.joinDocs<_post, _user>({
119
+ $leftCollection: "posts",
120
+ $rightCollection: "users",
121
+ $mode: "inner", // "inner" | "left" | "right" | "outer"
122
+ $on: { userId: { $eq: "id" } },
123
+ $select: ["title", "name"],
124
+ $limit: 50
125
+ })
126
+ ```
127
+
128
+ ### Real-Time Streaming
129
+
130
+ ```typescript
131
+ // Stream new/updated documents
132
+ for await (const doc of Fylo.findDocs<_user>("users")) {
133
+ console.log(doc)
134
+ }
135
+
136
+ // Stream deletions
137
+ for await (const _id of Fylo.findDocs<_user>("users").onDelete()) {
138
+ console.log("deleted:", _id)
139
+ }
140
+
141
+ // Watch a single document
142
+ for await (const doc of Fylo.getDoc<_user>("users", _id)) {
143
+ console.log(doc)
144
+ }
145
+ ```
146
+
147
+ ### Bulk Import / Export
148
+
149
+ ```typescript
150
+ const fylo = new Fylo()
151
+
152
+ // Import from JSON array or NDJSON URL
153
+ const count = await fylo.importBulkData<_user>("users", new URL("https://example.com/users.json"), 1000)
154
+
155
+ // Export all documents
156
+ for await (const doc of Fylo.exportBulkData<_user>("users")) {
157
+ console.log(doc)
158
+ }
159
+ ```
160
+
161
+ ### Rollback
162
+
163
+ Every write is tracked as a transaction. If a batch write partially fails, Fylo automatically rolls back. You can also trigger it manually:
164
+
165
+ ```typescript
166
+ const fylo = new Fylo()
167
+ await fylo.putData("users", { name: "test" })
168
+ await fylo.rollback() // undoes all writes in this instance
169
+ ```
170
+
171
+ ### CLI
172
+
173
+ ```bash
174
+ fylo.query "SELECT * FROM users WHERE age > 25 LIMIT 10"
175
+ ```
176
+
177
+ ### Schema Validation
178
+
179
+ When `STRICT` is set, documents are validated against CHEX schemas before writes:
180
+
181
+ ```bash
182
+ STRICT=true bun run start
183
+ ```
184
+
185
+ Schemas are `.d.ts` interface declarations generated by [`@vyckr/chex`](https://github.com/vyckr/chex).
186
+
187
+ ## Development
188
+
189
+ ```bash
190
+ bun test # Run all tests
191
+ bun run build # Compile TypeScript
192
+ bun run typecheck # Type-check without emitting
193
+ bun run lint # ESLint
194
+ ```
195
+
196
+ ### Local S3 (LocalStack)
197
+
198
+ ```bash
199
+ docker compose up aws
200
+ ```
201
+
202
+ This starts LocalStack on `localhost:4566`. Set `S3_ENDPOINT=http://localhost:4566` to route S3 calls locally.
203
+
204
+ ## Security
205
+
206
+ ### What Fylo does NOT provide
207
+
208
+ Fylo is a low-level storage abstraction. The following must be implemented by the integrating application:
209
+
210
+ - **Authentication** — Fylo has no concept of users or sessions. Any caller with access to the Fylo instance can read and write any collection.
211
+ - **Authorization** — `executeSQL` and all document operations accept any collection name with no permission check. In multi-tenant applications, a caller can access any collection unless the integrator enforces a boundary above Fylo.
212
+ - **Rate limiting** — There is no built-in request throttling. An attacker with access to the instance can flood S3 with requests or trigger expensive operations without restriction. Add rate limiting and document-size limits in your service layer.
213
+
214
+ ### Secure configuration
215
+
216
+ | Concern | Guidance |
217
+ |---------|----------|
218
+ | AWS credentials | Never commit credentials to version control. Use IAM instance roles or inject via CI secrets. Rotate any credentials that have been exposed. |
219
+ | `ENCRYPTION_KEY` | Must be at least 32 characters. Use a high-entropy random value. |
220
+ | `CIPHER_SALT` | Set a unique random value per deployment to prevent cross-instance precomputation attacks. |
221
+ | `REDIS_URL` | Always set explicitly. Use `rediss://` (TLS) in production with authentication credentials in the URL. |
222
+ | Collection names | Must match `^[a-z0-9][a-z0-9\-]*[a-z0-9]$`. Names are validated before any shell or S3 operation. |
223
+
224
+ ### Encrypted fields
225
+
226
+ Fields listed in `$encrypted` in a collection schema are encrypted with AES-256-CBC. By default a random IV is used per write (non-deterministic). Pass `deterministic: true` to `Cipher.encrypt()` only for fields that require `$eq`/`$ne` queries — deterministic encryption leaks value equality to observers of stored ciphertext.
227
+
228
+ ## License
229
+
230
+ MIT