bare-ripgrep 0.0.0-stage → 0.0.1

package/README.md

@@ -1,3 +1,95 @@
-# Temporary Holding Version
+# bare-ripgrep
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+ripgrep's search and walk crates (`ignore`, `grep-searcher`, `grep-regex`) as a native addon for Bare and Node. It searches a local folder without an `rg` subprocess, so it also runs on iOS and Android.
+
+The result is meant to match the `grep` tool of qvac-harness (`lib/tools/fs/search.ts`, `walk.ts`, `ignore.ts`, `glob.ts`) while being faster and not stopping at its 10k-entry JS walk cap.
+
+## Install
+
+```
+npm i bare-ripgrep
+```
+
+## Usage
+
+```js
+const { search, walk } = require('bare-ripgrep')
+
+const { hits, truncated, cut } = await search({
+  root: '/path/to/project',
+  queries: ['todo', 'fixme'],
+  include: '*.ts'
+})
+// hits: [{ path: 'src/a.ts', line: 12, text: '// TODO: ...' }]
+
+const { files } = await walk({ root: '/path/to/project', folder: 'src' })
+// files: [{ path: 'src/a.ts', mtime: 1760000000000 }]
+```
+
+### `search(options)`
+
+| Option           | Default | Meaning                                                                                   |
+| ---------------- | ------- | ----------------------------------------------------------------------------------------- |
+| `root`           |         | Absolute directory. Returned paths are relative to it, `/`-separated.                     |
+| `folder`         | `''`    | Subfolder of `root` to search.                                                            |
+| `queries`        |         | Each query is an alternative. Literal text unless `regex`.                                |
+| `regex`          | `false` | Queries are regular expressions (Rust `regex` syntax).                                    |
+| `caseSensitive`  | `false` |                                                                                           |
+| `include`        |         | Glob or array of globs. A bare name matches the basename at any depth; with `/` the path. |
+| `deny`           |         | Absolute paths whose files and folders are skipped, case-insensitively.                   |
+| `includeIgnored` | `false` | Also walk what `.gitignore`, `.ignore` and the hidden tool folders leave out.             |
+| `maxMatches`     | `1000`  | Most hits returned.                                                                       |
+| `maxEntries`     | `10000` | Most entries (files and folders) the walk keeps.                                          |
+| `signal`         |         | `AbortSignal`. Abort rejects with an `AbortError` and stops the native work.              |
+
+Resolves `{ hits: { path, line, text }[], truncated, cut }`. Hits are sorted by path, then line. `truncated` is set when there were more hits than `maxMatches` or more entries than `maxEntries`. `cut` is set when a line past 1 MiB was searched in its head only.
+
+### `walk(options)`
+
+Same scope options (`root`, `folder`, `include`, `deny`, `includeIgnored`, `maxEntries`, `signal`). Resolves `{ files: { path, mtime }[], truncated }`, sorted by path; `mtime` is in milliseconds.
+
+## Semantics
+
+- Case-insensitive by default. Queries are literal unless `regex`.
+- A file with a NUL in its first 1024 bytes is binary and skipped; a later NUL is text.
+- `.gitignore` and `.ignore` are honoured with or without a git repository, `.ignore` wins. Ignore files between `root` and `folder` apply. `.git/info/exclude`, global ignore files and ignore files above `root` are not read. Dotfiles are searched.
+- `.git`, `node_modules` and `.trash` are always skipped. `.hg`, `.svn`, `.venv`, `.cache` and `.next` are skipped unless `includeIgnored`.
+- Symlinks are never followed, files or folders.
+- Hit text is the line without its `\n` (a `\r` stays), cut to 500 UTF-16 units. A line past 1 MiB is searched in its head only.
+- With `maxMatches`, the hits kept are the first by path then line, whatever the thread timing.
+
+## Differences from the harness engine
+
+- **Regex dialect.** Queries with `regex` use Rust's `regex` syntax, not JavaScript's: no look-around and no backreferences (they reject with `invalid regex`), `\w`, `\d` and `\b` are Unicode-aware, a pattern naming a newline is rejected, and `.` also matches `\r`. Literal queries are unaffected.
+- **Case folding.** `caseSensitive: false` uses Unicode simple case folding, which can differ from JS `i` on a few characters.
+- **Ignore engine.** Ignore rules are git's, through the `ignore` crate, not the harness's regex translation; edge cases of exotic patterns can differ. A `folder` that an ancestor's ignore file ignores yields nothing; the harness searches it.
+- **Caps under parallelism.** The walk is parallel. When `maxEntries` cuts it short, which entries were kept can differ between runs. The `maxMatches` result is deterministic. Entries under a `deny` path are pruned, not counted.
+- **Long lines.** A line past 1 MiB is searched in its head only and sets `cut` (the harness sets `cut` only for a cut drive read). A line past 16 MiB ends that file's scan, keeps its earlier hits and sets `cut`.
+- **Text clipping.** A surrogate pair split by the 500-unit cut is dropped whole; JS would keep a lone surrogate.
+- **Not provided.** `maxMs` and `skippedLinks`. A file that cannot be opened is skipped silently.
+
+## Build
+
+`bare-make generate && bare-make build && bare-make install` (`npm run build`) writes `prebuilds/<platform>-<arch>/bare-ripgrep.bare` and `bare-ripgrep.node`: one build, one C binding (`binding.c`, written against `js.h`), loaded by Bare directly and by Node through `bare-compat-napi`.
+
+The Rust crate (`Cargo.toml`, `src/`) builds as a `staticlib` through `cmake-cargo`, which runs `cargo build --release --target <triple>` and imports the archive. It needs `cargo` and the rustup target of the platform being built. The C side only passes a JSON request to Rust and a JSON result back; the work runs on the libuv thread pool (`uv_queue_work`), and the walk fans out over its own threads. The Rust entry points never unwind into C.
+
+| Target                                     | Rust triple                                             |
+| ------------------------------------------ | ------------------------------------------------------- |
+| `darwin-arm64`, `darwin-x64`               | `aarch64-apple-darwin`, `x86_64-apple-darwin`           |
+| `linux-arm64`, `linux-x64`                 | `aarch64-unknown-linux-gnu`, `x86_64-unknown-linux-gnu` |
+| `win32-arm64`, `win32-x64`                 | `aarch64-pc-windows-msvc`, `x86_64-pc-windows-msvc`     |
+| `ios-arm64`                                | `aarch64-apple-ios`                                     |
+| `ios-arm64-simulator`, `ios-x64-simulator` | `aarch64-apple-ios-sim`, `x86_64-apple-ios`             |
+| `android-arm64`, `android-x64`             | `aarch64-linux-android`, `x86_64-linux-android`         |
+
+## Test and bench
+
+```
+npm test          # brittle, on Node and on Bare
+node bench/index.js <dir> [pattern...]   # search() against `rg -i -F` under the same ignore rules
+```
+
+## License
+
+Apache-2.0

package/binding.c

@@ -0,0 +1,169 @@
+#include <assert.h>
+#include <bare.h>
+#include <js.h>
+#include <stdbool.h>
+#include <stdlib.h>
+#include <string.h>
+#include <uv.h>
+
+#include "src/bare_ripgrep.h"
+
+typedef struct {
+  uv_work_t handle;
+  js_env_t *env;
+  js_ref_t *cb;
+  js_deferred_teardown_t *teardown;
+  rg_job_t *job;
+  bool exiting;
+} bare_ripgrep_job_t;
+
+static void
+bare_ripgrep__on_work (uv_work_t *handle) {
+  bare_ripgrep_job_t *job = handle->data;
+
+  rg_job_run(job->job);
+}
+
+static void
+bare_ripgrep__on_after_work (uv_work_t *handle, int status) {
+  int err;
+
+  bare_ripgrep_job_t *job = handle->data;
+  js_env_t *env = job->env;
+  js_deferred_teardown_t *teardown = job->teardown;
+
+  if (!job->exiting) {
+    js_handle_scope_t *scope;
+    err = js_open_handle_scope(env, &scope);
+    assert(err == 0);
+
+    js_value_t *cb;
+    err = js_get_reference_value(env, job->cb, &cb);
+    assert(err == 0);
+
+    size_t len;
+    const char *json = rg_job_result(job->job, &len);
+
+    js_value_t *argv[1];
+    err = js_create_string_utf8(env, (const utf8_t *) json, len, &argv[0]);
+    assert(err == 0);
+
+    js_value_t *global;
+    err = js_get_global(env, &global);
+    assert(err == 0);
+
+    js_call_function_with_checkpoint(env, global, cb, 1, argv, NULL);
+
+    err = js_close_handle_scope(env, scope);
+    assert(err == 0);
+  }
+
+  err = js_delete_reference(env, job->cb);
+  assert(err == 0);
+
+  rg_job_free(job->job);
+  free(job);
+
+  err = js_finish_deferred_teardown_callback(teardown);
+  assert(err == 0);
+}
+
+static void
+bare_ripgrep__on_teardown (js_deferred_teardown_t *handle, void *data) {
+  bare_ripgrep_job_t *job = data;
+
+  job->exiting = true;
+
+  rg_job_cancel(job->job);
+}
+
+static js_value_t *
+bare_ripgrep_start (js_env_t *env, js_callback_info_t *info) {
+  int err;
+
+  size_t argc = 2;
+  js_value_t *argv[2];
+
+  err = js_get_callback_info(env, info, &argc, argv, NULL, NULL);
+  assert(err == 0);
+
+  assert(argc == 2);
+
+  size_t len;
+  err = js_get_value_string_utf8(env, argv[0], NULL, 0, &len);
+  assert(err == 0);
+
+  utf8_t *json = malloc(len + 1);
+  err = js_get_value_string_utf8(env, argv[0], json, len + 1, &len);
+  assert(err == 0);
+
+  bare_ripgrep_job_t *job = malloc(sizeof(bare_ripgrep_job_t));
+  job->env = env;
+  job->exiting = false;
+  job->handle.data = job;
+  job->job = rg_job_new((const char *) json, len);
+
+  free(json);
+
+  err = js_create_reference(env, argv[1], 1, &job->cb);
+  assert(err == 0);
+
+  uv_loop_t *loop;
+  err = js_get_env_loop(env, &loop);
+  assert(err == 0);
+
+  err = js_add_deferred_teardown_callback(env, bare_ripgrep__on_teardown, job, &job->teardown);
+  assert(err == 0);
+
+  err = uv_queue_work(loop, &job->handle, bare_ripgrep__on_work, bare_ripgrep__on_after_work);
+  assert(err == 0);
+
+  js_value_t *handle;
+  err = js_create_external(env, job, NULL, NULL, &handle);
+  assert(err == 0);
+
+  return handle;
+}
+
+static js_value_t *
+bare_ripgrep_cancel (js_env_t *env, js_callback_info_t *info) {
+  int err;
+
+  size_t argc = 1;
+  js_value_t *argv[1];
+
+  err = js_get_callback_info(env, info, &argc, argv, NULL, NULL);
+  assert(err == 0);
+
+  assert(argc == 1);
+
+  bare_ripgrep_job_t *job;
+  err = js_get_value_external(env, argv[0], (void **) &job);
+  assert(err == 0);
+
+  rg_job_cancel(job->job);
+
+  return NULL;
+}
+
+static js_value_t *
+bare_ripgrep_exports (js_env_t *env, js_value_t *exports) {
+  int err;
+
+#define V(name, fn) \
+  { \
+    js_value_t *val; \
+    err = js_create_function(env, name, -1, fn, NULL, &val); \
+    assert(err == 0); \
+    err = js_set_named_property(env, exports, name, val); \
+    assert(err == 0); \
+  }
+
+  V("start", bare_ripgrep_start)
+  V("cancel", bare_ripgrep_cancel)
+#undef V
+
+  return exports;
+}
+
+BARE_MODULE(bare_ripgrep, bare_ripgrep_exports)

package/binding.js

@@ -0,0 +1,3 @@
+require.addon = require('require-addon')
+
+module.exports = require.addon('.', __filename)

package/index.d.ts

@@ -0,0 +1,70 @@
+export interface Hit {
+  /** Relative to `root`, `/`-separated. */
+  path: string
+  /** 1-based. */
+  line: number
+  /** The line without its newline, at most 500 UTF-16 units. */
+  text: string
+}
+
+export interface WalkedFile {
+  /** Relative to `root`, `/`-separated. */
+  path: string
+  /** Milliseconds since the epoch. */
+  mtime: number
+}
+
+interface Scope {
+  /** Absolute directory every returned path is relative to. */
+  root: string
+  /** Subfolder of `root` to search, `''` for all of it. */
+  folder?: string
+  /**
+   * Glob a file must match: a bare name matches the basename at any depth, a pattern with
+   * `/` matches the path relative to `root`. Several globs are alternatives.
+   */
+  include?: string | string[]
+  /** Absolute paths whose files and folders are skipped (case-insensitive). */
+  deny?: string[]
+  /** Also walk what ignore files and the hidden tool folders leave out. */
+  includeIgnored?: boolean
+  /** Most entries (files and folders) the walk keeps, default 10000. */
+  maxEntries?: number
+  signal?: AbortSignal
+}
+
+export interface SearchOptions extends Scope {
+  /** Each query is an alternative. */
+  queries: string[]
+  /** Queries are regular expressions rather than literals. */
+  regex?: boolean
+  caseSensitive?: boolean
+  /** Most hits returned, default 1000. */
+  maxMatches?: number
+}
+
+export interface SearchResult {
+  /** Sorted by path, then line. */
+  hits: Hit[]
+  /** More hits or entries than the caps allowed. */
+  truncated: boolean
+  /** A line past 1 MiB was searched in its head only. */
+  cut: boolean
+}
+
+export interface WalkOptions extends Scope {}
+
+export interface WalkResult {
+  /** Sorted by path. */
+  files: WalkedFile[]
+  truncated: boolean
+}
+
+export class AbortError extends Error {
+  name: 'AbortError'
+  code: 'ABORT_ERR'
+}
+
+export function search(options: SearchOptions): Promise<SearchResult>
+
+export function walk(options: WalkOptions): Promise<WalkResult>

package/index.js

@@ -0,0 +1,100 @@
+const binding = require('./binding')
+
+const MAX_MATCHES = 1000
+const MAX_ENTRIES = 10_000
+
+class AbortError extends Error {
+  constructor() {
+    super('The operation was aborted')
+    this.name = 'AbortError'
+    this.code = 'ABORT_ERR'
+  }
+}
+
+exports.AbortError = AbortError
+
+exports.search = function search({
+  root,
+  folder = '',
+  queries,
+  regex = false,
+  caseSensitive = false,
+  include,
+  deny,
+  includeIgnored = false,
+  maxMatches = MAX_MATCHES,
+  maxEntries = MAX_ENTRIES,
+  signal
+}) {
+  return run(
+    {
+      op: 'search',
+      root,
+      folder,
+      queries,
+      regex,
+      caseSensitive,
+      include: list(include),
+      deny: list(deny),
+      includeIgnored,
+      maxMatches: count(maxMatches),
+      maxEntries: count(maxEntries)
+    },
+    signal
+  )
+}
+
+exports.walk = function walk({
+  root,
+  folder = '',
+  include,
+  deny,
+  includeIgnored = false,
+  maxEntries = MAX_ENTRIES,
+  signal
+}) {
+  return run(
+    {
+      op: 'walk',
+      root,
+      folder,
+      include: list(include),
+      deny: list(deny),
+      includeIgnored,
+      maxMatches: 0,
+      maxEntries: count(maxEntries)
+    },
+    signal
+  )
+}
+
+function run(request, signal) {
+  return new Promise((resolve, reject) => {
+    if (signal && signal.aborted) return reject(new AbortError())
+
+    let job = null
+
+    const onabort = () => {
+      binding.cancel(job)
+      reject(new AbortError())
+    }
+
+    job = binding.start(JSON.stringify(request), (json) => {
+      if (signal) signal.removeEventListener('abort', onabort)
+
+      const result = JSON.parse(json)
+      if (result.error) reject(new Error(result.error))
+      else resolve(result)
+    })
+
+    if (signal) signal.addEventListener('abort', onabort, { once: true })
+  })
+}
+
+function list(value) {
+  return value === undefined ? [] : [].concat(value)
+}
+
+function count(value) {
+  return Math.min(value, Number.MAX_SAFE_INTEGER)
+}

package/package.json

@@ -1,6 +1,68 @@
 {
   "name": "bare-ripgrep",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.0.1",
+  "description": "ripgrep's search and walk crates as a native addon for Bare and Node",
+  "main": "index.js",
+  "exports": {
+    "./package": "./package.json",
+    ".": {
+      "types": "./index.d.ts",
+      "default": "./index.js"
+    }
+  },
+  "files": [
+    "package.json",
+    "index.js",
+    "index.d.ts",
+    "binding.js",
+    "binding.c",
+    "Cargo.toml",
+    "Cargo.lock",
+    "src",
+    "CMakeLists.txt",
+    "NOTICE.md",
+    "prebuilds"
+  ],
+  "addon": true,
+  "scripts": {
+    "build": "bare-make generate && bare-make build && bare-make install",
+    "bench": "node bench/index.js",
+    "format": "prettier . --write",
+    "test": "npm run test:node && npm run test:bare",
+    "test:node": "brittle-node test/all.mjs",
+    "test:bare": "brittle-bare test/all.mjs",
+    "test:generate": "brittle-make-test test/all.mjs test/*.test.js",
+    "lint": "prettier . --check && lunte"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/tetherto/bare-ripgrep.git"
+  },
+  "author": "Tether Inc",
+  "license": "Apache-2.0",
+  "bugs": {
+    "url": "https://github.com/tetherto/bare-ripgrep/issues"
+  },
+  "homepage": "https://github.com/tetherto/bare-ripgrep",
+  "engines": {
+    "bare": ">=1.16.0"
+  },
+  "dependencies": {
+    "require-addon": "^1.1.0"
+  },
+  "devDependencies": {
+    "bare-abort-controller": "^1.0.0",
+    "bare-compat-napi": "^1.3.5",
+    "bare-fs": "^4.1.2",
+    "bare-make": "^1.7.0",
+    "bare-os": "^3.9.4",
+    "bare-path": "^3.0.0",
+    "brittle": "^4.1.0",
+    "cmake-bare": "^1.8.0",
+    "cmake-cargo": "^0.1.1",
+    "cmake-napi": "^1.2.1",
+    "lunte": "^1.8.4",
+    "prettier": "^3.9.6",
+    "prettier-config-holepunch": "^2.0.0"
+  }
+}

package/src/bare_ripgrep.h

@@ -0,0 +1,23 @@
+#ifndef BARE_RIPGREP_H
+#define BARE_RIPGREP_H
+
+#include <stddef.h>
+
+typedef struct rg_job_s rg_job_t;
+
+rg_job_t *
+rg_job_new (const char *json, size_t len);
+
+void
+rg_job_run (const rg_job_t *job);
+
+void
+rg_job_cancel (const rg_job_t *job);
+
+const char *
+rg_job_result (const rg_job_t *job, size_t *len);
+
+void
+rg_job_free (rg_job_t *job);
+
+#endif

package/src/glob.rs

@@ -0,0 +1,47 @@
+use regex::Regex;
+
+/// Mirrors qvac-harness `compileGlob`: `*` and `?` stay inside a segment, `**/` spans
+/// directories, anything else is literal. A pattern with `/` tests the path, else the name.
+pub struct Glob {
+    regex: Regex,
+    on_path: bool,
+}
+
+impl Glob {
+    pub fn new(pattern: &str) -> Result<Glob, String> {
+        let pattern = pattern.strip_prefix("./").unwrap_or(pattern);
+        let chars: Vec<char> = pattern.chars().collect();
+        let mut source = String::from("(?s)^");
+        let mut i = 0;
+        while i < chars.len() {
+            let rest = &chars[i..];
+            if rest.starts_with(&['*', '*', '/']) {
+                source.push_str("(?:.*/)?");
+                i += 3;
+            } else if rest.starts_with(&['*', '*']) {
+                source.push_str(".*");
+                i += 2;
+            } else if rest[0] == '*' {
+                source.push_str("[^/]*");
+                i += 1;
+            } else if rest[0] == '?' {
+                source.push_str("[^/]");
+                i += 1;
+            } else {
+                source.push_str(&regex::escape(&rest[0].to_string()));
+                i += 1;
+            }
+        }
+        source.push('$');
+
+        let regex = Regex::new(&source).map_err(|err| format!("invalid include glob: {err}"))?;
+        Ok(Glob {
+            regex,
+            on_path: pattern.contains('/'),
+        })
+    }
+
+    pub fn matches(&self, name: &str, path: &str) -> bool {
+        self.regex.is_match(if self.on_path { path } else { name })
+    }
+}