@infino-ai/infino 0.1.0

package/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

package/README.md

@@ -0,0 +1,348 @@
+# infino
+
+**SQL, full-text, and vector search over your data on object storage — one engine, no server to run.**
+
+Infino keeps your data in Apache Parquet on object storage (local disk, Amazon
+S3, or any S3-compatible store) and runs SQL, full-text (BM25), and vector
+search over it from a single system. Each file is a valid Parquet file with BM25
+and vector indexes embedded directly inside it; a table composes many such files
+with snapshot-isolated reads, append-only writes, and atomic commits. It runs in
+your process — there is no daemon, no cluster, and no managed service to operate.
+
+Use it for **RAG**, **agent memory**, **hybrid search**, and **semantic search**:
+it's an embedded **vector database**, **full-text (BM25)** search engine, and
+**SQL** query engine in one library — no separate vector database or search
+server to run.
+
+Synchronous, Arrow at the boundary: pass arrays of objects (or apache-arrow
+`Table`s) in, get plain records out; pass `{ arrow: true }` to a search or query
+for an apache-arrow `Table` instead.
+
+## Install
+
+```sh
+npm install @infino-ai/infino
+```
+
+A prebuilt native binary is selected automatically at install time — no Rust
+toolchain required. Supported platforms:
+
+| Platform              | Architectures |
+| --------------------- | ------------- |
+| macOS                 | x64, arm64    |
+| Linux (glibc)         | x64, arm64    |
+| Linux (musl / Alpine) | x64, arm64    |
+
+`apache-arrow` is installed as a dependency and used at the boundary (passing in
+`Table`s, or `{ arrow: true }` results). Requires Node.js >= 18.
+
+## Quickstart
+
+```javascript
+import { connect, IndexSpec } from "@infino-ai/infino";
+
+// Connect to a catalog. Use a local path or an S3 URI for durable storage;
+// "memory://" is ephemeral and handy for tests.
+const db = connect("./data");
+
+// Tiny stand-in for your embedding model so this runs as-is — a 16-dim
+// one-hot by topic. Real embeddings are dense and higher-dimensional.
+const embed = (topic) => { const v = Array(16).fill(0.0); v[topic] = 1.0; return v; };
+
+// Declare a schema and which columns to index. An `_id` column is added
+// automatically — you don't define it.
+const docs = db.createTable(
+  "docs",
+  { source: "large_utf8", body: "large_utf8", embedding: { vector: 16 } },
+  new IndexSpec().fts("body").vector("embedding", 16, 1, "cosine"),
+);
+
+// Append rows. One append is one atomic commit.
+docs.append([
+  { source: "help-center", body: "To cancel a subscription, open Settings then Billing.", embedding: embed(0) },
+  { source: "help-center", body: "Refunds return to the original payment method.",         embedding: embed(0) },
+  { source: "blog",        body: "Enable dark mode under Settings then Appearance.",        embedding: embed(1) },
+]);
+
+// Three ways to retrieve context to ground an agent's next answer:
+const keyword  = docs.bm25Search("body", "cancel subscription", 5);            // BM25
+const semantic = docs.vectorSearch("embedding", embed(0), 5);                  // vector kNN
+const billing  = db.querySql("SELECT body FROM docs WHERE source = 'help-center'");  // SQL filter
+```
+
+CommonJS works too — `const { connect, IndexSpec } = require("@infino-ai/infino");`.
+
+## Examples
+
+Runnable, end-to-end examples in [`examples/`](examples) (each its own folder
+with a README; build the addon first, then `npm install && node index.mjs`):
+
+- [`agent-memory/`](examples/agent-memory) — infino as an AI agent's long-term
+  memory: load a real multi-session conversation, then recall it with hybrid
+  search, query it with SQL (`GROUP BY`, filters), and forget parts of it.
+- [`hybrid-search-api/`](examples/hybrid-search-api) — an embedded HTTP search
+  service over a real product catalog, ranked by native `hybrid_search` — with
+  no separate search server to run.
+
+## Core concepts
+
+- **Connection** — a handle to a catalog (a set of tables under one URI). Open
+  it with `connect(uri)`.
+- **Table** — an append-only, snapshot-isolated collection of rows. Each table
+  carries an auto-generated `_id` column.
+- **IndexSpec** — declares which columns are full-text (BM25) and which are
+  vector indexed. Columns without an index are still stored, filterable in SQL,
+  and returnable via projection.
+- **Commits** — every `append`, `update`, and `delete` is a single atomic
+  commit. Readers see a consistent snapshot and are never torn by a concurrent
+  write.
+- **Arrow at the boundary** — searches return plain records (or an apache-arrow
+  `Table` with `{ arrow: true }`); `append` and `update` accept an array of
+  objects or an apache-arrow `Table` / `RecordBatch`.
+
+## Full-text search
+
+```javascript
+const docs = db.createTable("docs", { title: "large_utf8" }, new IndexSpec().fts("title"));
+docs.append([{ title: "the quick brown fox" }, { title: "a lazy dog" }]);
+
+// Ranked BM25 — higher score is a better match.
+docs.bm25Search("title", "quick fox", 10);                  // OR by default
+docs.bm25Search("title", "quick fox", 10, { mode: "and" }); // require all terms
+
+// Unranked matching (score is 0): every row containing the term(s),
+// or an exact whole-value match.
+docs.tokenMatch("title", "fox");
+docs.exactMatch("title", "the quick brown fox");
+```
+
+## Vector search
+
+Vector columns are `FixedSizeList<Float32, dim>` with `dim` in `[16, 4096]`. The
+distance metric is fixed when you declare the index (`"cosine"`, `"l2sq"`, or
+`"negdot"`); for vector results a smaller score is nearer. The query vector is a
+`number[]` or `Float32Array`.
+
+```javascript
+const spec = new IndexSpec().vector("emb", 384, 256, "cosine"); // (column, dim, nCent, metric)
+const vecs = db.createTable("vecs", { emb: { vector: 384 } }, spec);
+
+vecs.vectorSearch("emb", queryVector, 10);                    // top-10 nearest
+vecs.vectorSearch("emb", queryVector, 10, { nprobe: 32 });    // probe more partitions (recall)
+vecs.vectorSearch("emb", queryVector, 10, { rerankMult: 4 }); // wider exact-rerank pool (recall)
+```
+
+**Filtered vector search.** Restrict the kNN to rows matching a text predicate —
+a pushdown *pre-filter*, so you get the nearest *matching* rows (not a
+post-filter over the global top-k). The filter `column` must be FTS-indexed.
+
+```javascript
+vecs.vectorSearch("emb", queryVector, 10, {
+  filter: { column: "title", query: "billing", mode: "or" },
+});
+```
+
+## Hybrid search
+
+Combine BM25 and vector search in **one query** with the `hybrid_search` table
+function — a single pass over both indexes, fused inside the engine (no separate
+reranker service, no two round-trips). Keyword-only search misses paraphrases;
+vector-only search misses exact terms — hybrid gets both. Results come back
+best-first with a fused `score`.
+
+```javascript
+const spec = new IndexSpec().fts("body").vector("emb", 384, 256, "cosine");
+const docs = db.createTable("docs", { body: "large_utf8", emb: { vector: 384 } }, spec);
+docs.append([{ body: "To cancel a subscription, open Settings then Billing.", emb: embed(/* … */) }]);
+
+// hybrid_search(table, text_col, query_text, vec_col, query_vec, k)
+const qvec = embed("how do I stop my plan?").join(",");
+db.querySql(
+  `SELECT _id, score FROM hybrid_search('docs', 'body', 'cancel subscription', 'emb', '${qvec}', 10)`,
+);
+```
+
+For a complete, runnable hybrid search service see the
+[`hybrid-search-api` example](examples/hybrid-search-api).
+
+## SQL
+
+Run SQL across the catalog's tables for analytics and filtering; the search
+functions are also available as SQL table functions. Results come back as plain
+records (or an apache-arrow `Table` with `{ arrow: true }`).
+
+```javascript
+db.querySql("SELECT COUNT(*) AS n FROM docs");
+db.querySql("SELECT title FROM docs WHERE title = 'a lazy dog'");
+
+// The search methods are also SQL table functions — bm25_search, vector_search,
+// and hybrid_search (see "Hybrid search" above) — so you can filter, join, and
+// aggregate over search results.
+db.querySql("SELECT _id, score FROM bm25_search('docs', 'title', 'fox', 10)");
+```
+
+## Projections
+
+By default a search returns just `_id` and `score` — no row data is decoded.
+Name the columns you want to materialize:
+
+```javascript
+docs.bm25Search("title", "fox", 10);                                   // _id + score only
+docs.bm25Search("title", "fox", 10, { projection: ["_id", "title", "score"] });
+```
+
+## Updates and deletes
+
+Mutations require durable storage (a local path or object store, not
+`memory://`). The predicate is a SQL boolean expression — the same thing you'd
+write after `WHERE` — evaluated against the table's columns.
+
+```javascript
+docs.append([{ title: "draft post" }, { title: "spam" }]);
+
+// Delete every row matching the predicate.
+docs.delete("title = 'spam'");
+
+// Replace matched rows 1:1 with new rows (same input shapes as append).
+const stats = docs.update("title = 'draft post'", [{ title: "published post" }]);
+console.log(stats.matched, stats.nTombstoned, stats.nNotFound);
+```
+
+`update` is a one-to-one replacement: the number of matched rows must equal the
+number you supply, otherwise it throws. Both methods return `{ matched,
+nTombstoned, nNotFound }`.
+
+## Optimize
+
+Many small appends produce many small files. `optimize` compacts them —
+merging small or underfilled files into larger ones — which keeps reads efficient.
+
+```javascript
+docs.optimize();                                                  // engine defaults
+docs.optimize({ targetSuperfileSizeMb: 256, minFillPercent: 50 });
+```
+
+## Storage backends
+
+`connect` selects the backend from the URI:
+
+| URI                   | Backend                                  |
+| --------------------- | ---------------------------------------- |
+| `./data`, `/abs/path` | Local filesystem                         |
+| `s3://bucket/prefix`  | Amazon S3 / S3-compatible object storage |
+| `memory://`           | In-process, ephemeral (testing)          |
+
+For S3-compatible stores that need an explicit endpoint and static credentials,
+pass them in `options` (omit to use ambient AWS credentials):
+
+```javascript
+const db = connect("s3://bucket/prefix", {
+  endpoint: "https://s3.example.com",
+  region: "us-east-1",
+  accessKey: "…",
+  secretKey: "…",
+});
+```
+
+### Local disk cache
+
+For object-storage-backed catalogs, a local disk cache keeps hot data on fast
+local storage. `coldFetchMode` controls how cache misses are served:
+`"hybrid_with_prefetch"`, `"range_only"`, or
+`"lazy_foreground_with_background_fill"`.
+
+```javascript
+const db = connect("s3://bucket/prefix", {
+  cacheDir: "/mnt/nvme/infino-cache",
+  cacheBudgetBytes: 64 * 1024 ** 3,
+  coldFetchMode: "lazy_foreground_with_background_fill",
+});
+```
+
+## Schema and type requirements
+
+- Full-text columns must be Arrow `LargeUtf8` (`"large_utf8"` in a descriptor).
+- Vector columns must be `FixedSizeList<Float32, dim>` (`{ vector: dim }`) with
+  `dim` in `[16, 4096]`.
+- The `_id` column is generated by the engine; do not declare it. It comes back
+  as a JavaScript `bigint`.
+- `createTable` accepts an apache-arrow `Schema` or a plain `{ column: type }`
+  descriptor; `append` / `update` accept an array of objects or an apache-arrow
+  `Table` / `RecordBatch`, coerced against the table's declared schema.
+
+## API reference
+
+- `connect(uri, options?)` — backend from the URI scheme. `options`:
+  S3-compatible credentials (`endpoint`, `region`, `accessKey`, `secretKey` —
+  `endpoint` requires the other three) and, for remote-backed tables, a local
+  disk cache (`cacheDir`, `cacheBudgetBytes`, `coldFetchMode`).
+- `Connection`
+  - `createTable(name, schema, IndexSpec)` / `openTable(name)` /
+    `dropTable(name, purge?)` (`purge = true` also deletes the data) /
+    `listTables()` / `querySql(sql, { arrow? })`.
+- `Table`
+  - `append(data)` — one `append` is one commit.
+  - `bm25Search(col, q, k, { mode?, projection?, arrow? })` — ranked BM25.
+  - `vectorSearch(col, query, k, { nprobe?, rerankMult?, filter?, projection?, arrow? })`
+    — ranked kNN; `filter` (`{ column, query, mode? }`, `column` FTS-indexed) is
+    a pushdown pre-filter.
+  - `tokenMatch(col, q, { mode?, projection?, arrow? })` /
+    `exactMatch(col, value, { projection?, arrow? })` — unranked (`score` is `0`).
+  - `update(predicate, data)` / `delete(predicate)` — mutate rows matching a SQL
+    predicate; return `{ matched, nTombstoned, nNotFound }`; require durable
+    storage.
+  - `optimize({ maxMemoryMb?, minFillPercent?, targetSuperfileSizeMb? })`.
+  - `schema()` — the table's apache-arrow `Schema`.
+- `IndexSpec().fts(col).vector(col, dim, nCent, metric)`.
+- `BUILDER_ID` (named export) — the engine's build identifier string.
+
+Search results default to `_id` + `score`; name columns in `projection` to
+materialize row data.
+
+## Building from source
+
+The binding is built with [napi-rs](https://napi.rs/). Building requires a Rust
+toolchain and access to crates.io.
+
+```sh
+cd infino-node
+npm install && npm run build && npm test
+```
+
+## Notes
+
+- The API is **synchronous**. In a long-running server, run calls in a
+  `worker_thread` so a query doesn't block the event loop.
+
+## FAQ
+
+**Is infino a vector database?** It does vector search, but it's more than that —
+an embedded engine that runs vector search *and* full-text (BM25) *and* SQL over
+one copy of your data. Reach for it wherever you'd use a vector database, plus the
+cases a vector store alone can't cover: keyword search, filtering, joins, and
+aggregates.
+
+**Does it need a server?** No. It runs in your Node.js process — no daemon, no
+cluster, no managed service. Your data is Parquet on local disk or S3.
+
+**Can it do hybrid (keyword + vector) search?** Yes, natively — BM25 and vector
+fused in a single pass via `hybrid_search` (see [Hybrid search](#hybrid-search)),
+not a client-side rerank.
+
+**Where is my data stored?** As Apache Parquet files on local disk or any
+S3-compatible object store; each file embeds its own BM25 and vector indexes.
+
+**Does it work with TypeScript?** Yes — the package ships type definitions and the
+API is identical from JavaScript and TypeScript. Both ESM `import` and CommonJS
+`require` work.
+
+**Do I need a Rust toolchain to install it?** No — a prebuilt native binary is
+selected automatically at install (macOS and Linux, x64 and arm64).
+
+**Is it a good fit for RAG or agent memory?** Yes, that's a primary use case:
+store documents or conversation history once, retrieve with hybrid search, and
+filter/aggregate with SQL. See the runnable [examples](examples).
+
+## License
+
+Apache-2.0.