@hitslop/slurp 0.0.0-stage → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.gitignore +2 -0
- package/CHANGELOG.md +40 -0
- package/LICENSE +21 -0
- package/README.md +229 -2
- package/binary-install.js +348 -0
- package/binary.js +124 -0
- package/install.js +4 -0
- package/npm-shrinkwrap.json +52 -0
- package/package.json +63 -3
- package/run-slurp.js +4 -0
package/.gitignore
ADDED
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.1.1
|
|
4
|
+
|
|
5
|
+
Run `slurp cloud upgrade` to put the fixes below on an existing Worker (your data and settings are kept).
|
|
6
|
+
|
|
7
|
+
- **RevenueCat webhooks (Preview)**: `slurp project webhook PROJECT revenuecat` gives the URL and the Authorization header value to paste into RevenueCat. Purchases, renewals, cancellations and refunds arrive as `revenuecat.<type>` events (retries are one row), and a `revenue` view in SQL lists them. RevenueCat's optional HMAC signing is not supported yet.
|
|
8
|
+
- **Fixed: a redelivered webhook could stop `slurp sync`.** A webhook with no time of its own was stamped with its arrival time, so a redelivery had the same id and a different time, which the importer treats as a conflict. Such a delivery now keeps the time it first arrived. A generic POST with no `Idempotency-Key` and no body `timestamp` is its own event each time. See [webhooks](docs/webhooks.md).
|
|
9
|
+
- **Fixed: archives stay unpublished after a long R2 outage.** Cloudflare retries a failing alarm six times and then stops. Publishing now schedules its own retries with backoff until R2 answers, so a quiet project publishes by itself once R2 is back (`ARCHIVE_RETRY_MS`).
|
|
10
|
+
- **Fixed: oversized bodies were buffered before being refused.** `/v1/events` now reads its body as a capped stream and stops at the limit, even with no `Content-Length`.
|
|
11
|
+
- **Fixed: an impossible timestamp such as `1e300` failed a whole Sentry envelope.** Sentry and Segment events with a time that is not a real date now fall back to `sent_at`, then the arrival time.
|
|
12
|
+
- Archive listings sign every URL with one key and one expiry, and report that expiry.
|
|
13
|
+
- **slurp.run measures itself with Slurp** (see `dogfood/`): page views, the feedback widget and two clicks go to a Worker in the maintainers' account, and the report that reads them is committed as an example. The site also got a mobile pass.
|
|
14
|
+
|
|
15
|
+
## 0.1.0
|
|
16
|
+
|
|
17
|
+
The first release.
|
|
18
|
+
|
|
19
|
+
- **Self-hosted event capture** in your own Cloudflare account: a Worker with one Durable Object per project (its keys, group commit, sealing alarm) and an R2 bucket of sealed gzip NDJSON archives, deployed with one command (`slurp cloud init`) or the Deploy to Cloudflare button.
|
|
20
|
+
- **Projects and senders**: a project is one event stream; a website, an app and a webhook can all send into it with their own publishable keys (`slurp project key create PROJECT --scope publishable --name ios`), and every event is stamped with its key's name as `sender` (a column in your local `events` table). A key names its project (`slurp_pk_<project>_<random>`), so a request never waits on a global lookup, a revoked key stops on the next request, and the SDKs need no project id. `slurp init` is idempotent: asking for a name that exists joins that project and never changes a key.
|
|
21
|
+
- **Python**: the Segment and Sentry Python SDKs (`segment-analytics-python`, `sentry-sdk`) work unmodified by changing the host or the DSN, verified against a real Worker on Python 3.9, 3.12 and 3.14. There is no native Python SDK.
|
|
22
|
+
- **Webhooks (Preview)**: GitHub, Stripe or any tool that can POST sends events to `/hooks/<provider>` on your Worker, each with its own named key (so events carry `sender = 'github'`), checked against the provider's signature and stored whole. `slurp project webhook PROJECT github` prints the URL, the secret and the steps; a `releases` view in SQL lists GitHub releases and deployments. Tested against the providers' documented signature examples, not yet live deliveries. See [webhooks](docs/webhooks.md).
|
|
23
|
+
- **`slurp` CLI** (macOS and Linux): `cloud init|status|doctor|upgrade|destroy`, `connect`, `setup`, `init`, `send`, `identify`, `sync` into a local DuckDB file, `sql`, `health`, `report` and agent skills. Install with `curl -LsSf https://slurp.run/install | sh` or `npm install -g @hitslop/slurp`.
|
|
24
|
+
- **JavaScript SDK**, `@hitslop/slurp-sdk`: browser and Node capture with a durable queue, batching and byte-identical retries (0.1.1 also runs on React Native).
|
|
25
|
+
- **Reports**: Markdown with named SQL, charts, metrics and tables, rendered to a self-contained HTML file (with hover details), PNG and PDF; themes and templates you can write yourself; DuckDB's behavioral functions for retention, funnels and sessions.
|
|
26
|
+
- **SQL helpers** in every project database: `events_with_person`, `calendar()`, `session_start()`, `pct()` and more.
|
|
27
|
+
- **React Native / Expo SDK**, `@hitslop/slurp-react-native`: events kept in AsyncStorage so they survive the app being killed, sent when the app goes to the background or the network returns, with device and app context. Preview: tested against fakes, not yet on a device.
|
|
28
|
+
- **Swift SDK** (iOS, macOS, tvOS, watchOS): Foundation only, file-backed durable queue, byte-identical retries, lifecycle flush. Add `https://github.com/hitSlop/slurp` in Xcode. Preview: unit tested and run against a local Worker, not yet in an app on a device.
|
|
29
|
+
- **Use your Segment SDK**: change its host to your Worker (`/v1/t|i|p|s|g|a`, `/v1/b`, `/v1/batch` and the settings document). Verified with analytics-node, analytics-next and analytics-swift.
|
|
30
|
+
- **Use your Sentry SDK**: change the DSN (the public key is your publishable key in hex). Errors are stored whole as `error` events and read with the new `errors` view. No symbolication or issue UI. Verified with `@sentry/node` and `@sentry/browser`.
|
|
31
|
+
- **Records up to 1 MiB**, stored whole and never truncated (was 32 KiB); a larger record is refused with `413`.
|
|
32
|
+
- **Sharper report PNGs**: drawn at 2x (1680 px wide for the 840 px layout), so text and charts stay crisp on high-density screens and when shared in chat or email.
|
|
33
|
+
- **`slurp mcp`**: an MCP server over stdio so Claude Code, Claude Desktop, Cursor, Codex and Windsurf can read your events (`describe_schema`, read-only `sql` in a sandbox with no local file access, `sync`, `health`, reports). `--allow-send` adds `send_event` and `identify`, tagged `context.source = "mcp"`.
|
|
34
|
+
- **`slurp notify slack|discord`**: post a message and a report image from the command line (Slack bot token or webhook, Discord webhook), for scheduled reports.
|
|
35
|
+
- **A small public API**: every method of the Worker's webrpc schema is tagged `@public` (a client outside this repo may use it) or `@internal` (CLI and control plane), so a client for third parties can be generated from the public methods only.
|
|
36
|
+
- **Link previews**: a social card (`og.png`) and Open Graph and Twitter tags on every page of the site.
|
|
37
|
+
- **Feedback widget** in the JS SDK (`@hitslop/slurp-sdk/feedback`): a drop-in "Feedback" button and form that sends a `feedback` event with the message, an optional rating and email.
|
|
38
|
+
- **One set of delivery rules in every SDK**, checked by the shared scenarios in `sdks/conformance/delivery.json`: any 2xx acknowledges a batch; 408, 429, 5xx, network errors and requests with no answer are retried with the identical bytes; 413 splits a batch and drops only a single oversized event; 401 and 403 stop the client; the queue is bounded in bytes as well as events (`maxQueueBytes`); damaged saved lines are dropped rather than sent; a replaced key starts sending again. React Native stores its queue in 256 KB pieces to stay under AsyncStorage's per-value limit, and Swift reports errors on a queue of its own so handlers can safely call back into the client.
|
|
39
|
+
- **One saved connection**: the CLI keeps one plain token per Worker (no logins, sessions or refresh), and `slurp whoami` (the `Auth.Whoami` RPC) says what any key is (admin, secret or publishable, and its project and name), so a bot can verify its own read key. `slurp connect` accepts only the admin token and says why. See [How the connection works](docs/connection.md).
|
|
40
|
+
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Longtail Labs
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,230 @@
|
|
|
1
|
-
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img src="docs/assets/slurp-mascot.png" width="300" alt="The Slurp anteater sipping a blue and red slushie through its long snout">
|
|
3
|
+
</p>
|
|
2
4
|
|
|
3
|
-
|
|
5
|
+
<h1 align="center"><img src="docs/assets/slurp-logo.png" width="170" alt="Slurp"></h1>
|
|
6
|
+
|
|
7
|
+
<p align="center"><a href="https://deploy.workers.cloudflare.com/?url=https%3A%2F%2Fgithub.com%2FhitSlop%2Fslurp%2Ftree%2Fmaster%2Fbackend"><img src="https://deploy.workers.cloudflare.com/button" alt="Deploy to Cloudflare"></a></p>
|
|
8
|
+
|
|
9
|
+
<p align="center"><b>Product analytics for coding agents, in your own Cloudflare account.</b></p>
|
|
10
|
+
<p align="center">Events in your Cloudflare account → <a href="https://duckdb.org"><img src="docs/assets/duckdb.jpeg" height="20" align="top" alt="DuckDB"></a> <a href="https://duckdb.org"><b>DuckDB</b></a> on your machine → your coding agent</p>
|
|
11
|
+
<p align="center">
|
|
12
|
+
<a href="https://github.com/hitSlop/slurp/actions/workflows/check.yml"><img src="https://github.com/hitSlop/slurp/actions/workflows/check.yml/badge.svg?branch=master" alt="CI status"></a>
|
|
13
|
+
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-e8002a" alt="MIT license"></a>
|
|
14
|
+
<a href="https://github.com/hitSlop/slurp/stargazers"><img src="https://img.shields.io/github/stars/hitSlop/slurp?style=flat&color=e8002a" alt="GitHub stars"></a>
|
|
15
|
+
<a href="https://discord.gg/jprRVWSRdn"><img src="https://img.shields.io/badge/Discord-join%20the%20chat-5865F2?logo=discord&logoColor=white" alt="Join the Discord"></a>
|
|
16
|
+
</p>
|
|
17
|
+
<!-- sdk-badges:start (generated by apps/site/scripts/readme-badges.mjs) -->
|
|
18
|
+
<p align="center">
|
|
19
|
+
<b>Capture from</b> <img src="https://img.shields.io/badge/JavaScript-SDK-F7DF1E?style=flat-square&labelColor=1f2327&logo=javascript&logoColor=white" alt="JavaScript: SDK">
|
|
20
|
+
<img src="https://img.shields.io/badge/TypeScript-types%20included-3178C6?style=flat-square&labelColor=1f2327&logo=typescript&logoColor=white" alt="TypeScript: types included">
|
|
21
|
+
<img src="https://img.shields.io/badge/Web-browser%20SDK-4285F4?style=flat-square&labelColor=1f2327&logo=googlechrome&logoColor=white" alt="Web: browser SDK">
|
|
22
|
+
<img src="https://img.shields.io/badge/Node.js-SDK-5FA04E?style=flat-square&labelColor=1f2327&logo=nodedotjs&logoColor=white" alt="Node.js: SDK">
|
|
23
|
+
<img src="https://img.shields.io/badge/React%20Native-preview%20SDK-61DAFB?style=flat-square&labelColor=1f2327&logo=react&logoColor=white" alt="React Native: preview SDK">
|
|
24
|
+
<img src="https://img.shields.io/badge/Swift-preview%20SDK-F05138?style=flat-square&labelColor=1f2327&logo=swift&logoColor=white" alt="Swift: preview SDK">
|
|
25
|
+
<img src="https://img.shields.io/badge/HTTP%20API-any%20language-073551?style=flat-square&labelColor=1f2327&logo=curl&logoColor=white" alt="HTTP API: any language"><br>
|
|
26
|
+
<b>Keep using</b> <img src="https://img.shields.io/badge/Segment-change%20the%20host-52BD94?style=flat-square&labelColor=1f2327&logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIwLjk3ZW0iIGhlaWdodD0iMWVtIiB2aWV3Qm94PSIwIDAgMjU2IDI2NSI%2BPHBhdGggZmlsbD0iIzRmYjU4YiIgZD0ibTIzMy41NiAxNDEuOTI3bC4xNy4wMTNsMTcuODkyIDEuODdhNC45MyA0LjkzIDAgMCAxIDMuMjI1IDEuNzA3bC4xMzMuMTYzbC0uMTcuMDg1YTQuOTMgNC45MyAwIDAgMSAxLjAyIDMuNzRhMTMzLjI3IDEzMy4yNyAwIDAgMS00MS42MDQgODEuMDgzYTEyOC44NiAxMjguODYgMCAwIDEtODcuNjI5IDM0LjM4YTEyNy41IDEyNy41IDAgMCAxLTQ2LjE1Ni04LjU3bC0uODAyLS4zMTJhNC43MiA0LjcyIDAgMCAxLTIuNjg2LTIuNTMzbC0uMDc3LS4xODdhNC45IDQuOSAwIDAgMS0uMDgzLTMuNjZsNy4wNjItMTcuMjNhNC44NDYgNC44NDYgMCAwIDEgNi4xMTgtMi43OTlsLjE2My4wNmMzNi4wOTcgMTMuOTM5IDc2Ljk4IDYuMDg5IDEwNS4zNDktMjAuMjI3YTEwNC40NiAxMDQuNDYgMCAwIDAgMzIuODkxLTYzLjMyYTQuOTMgNC45MyAwIDAgMSA1LjAxMy00LjI3em0tMTkwLjA4IDY0LjMxbC4yNTEtLjAwMmwuMjUzLjAwMmM4LjEyLjA5MyAxNC42NTggNi42NTkgMTQuNzQ2IDE0Ljc0OXYuMjUzbC0uMDAyLjI1MmMtLjE0MSA4LjI4NC02Ljk3IDE0Ljg4Ni0xNS4yNTQgMTQuNzQ1cy0xNC44ODUtNi45Ny0xNC43NDUtMTUuMjU0Yy4xMzktOC4xMTUgNi42OTUtMTQuNjE1IDE0Ljc1LTE0Ljc0NU00LjkzIDE0Ny4wODJoMTQ2LjMxNmE0Ljk3MyA0Ljk3MyAwIDAgMSA0LjkyOCA0Ljg0NGwuMDAyLjE3MXYxOC4zMTZhNC45NzQgNC45NzQgMCAwIDEtNC43NiA1LjAxbC0uMTcuMDA1SDQuOTNBNC45NzUgNC45NzUgMCAwIDEgMCAxNzAuNTg0di0xOC42NTlhNC45NzUgNC45NzUgMCAwIDEgNC43NTUtNC44Mzh6TTE2OS41NiA3LjMxMWE0Ljk3IDQuOTcgMCAwIDEgMi44NDggMi42MzVhNS4xIDUuMSAwIDAgMSAwIDMuODY3bC02LjM3NSAxNi45OTlhNC44NDUgNC44NDUgMCAwIDEtNi4xNjIgMi45NzRBMTAxLjIzIDEwMS4yMyAwIDAgMCA2Mi4xMyA1MS4yNTJhMTA1LjI3IDEwNS4yNyAwIDAgMC0zNC41MDcgNTQuOTlhNC45MyA0LjkzIDAgMCAxLTQuNzYgMy42OThoLTEuMTA1TDQuMjUgMTA1LjczM2E0LjkgNC45IDAgMCAxLTMuMTAzLTIuMjk1aC0uMDg1QTQuOTMgNC45MyAwIDAgMSAuNTEgOTkuNTdhMTMzLjQgMTMzLjQgMCAwIDEgNDQuNDEtNzAuMjA0Qzc5LjczOS43IDEyNy4wMTktNy42NjYgMTY5LjU2IDcuMzExbS02NC44MDcgNzMuNDM0SDI1MS4wN2E0Ljk3IDQuOTcgMCAwIDEgNC45MjIgNC42N2wuMDA4LjE3NHYxOC4zMTdhNC45NzMgNC45NzMgMCAwIDEtNC43NiA1LjAxbC0uMTcuMDA1SDEwNC43NTRhNC45NyA0Ljk3IDAgMCAxLTQuODg2LTQuODQybC0uMDAyLS4xNzNWODUuNzU5YTQuOTcgNC45NyAwIDAgMSA0LjcxNS01LjAwOHptMTAxLjU3Mi01NS44ODNsLjI1Mi0uMDAybC4yNTMuMDAyYzguMTIuMDkzIDE0LjY1OCA2LjY1OSAxNC43NDYgMTQuNzQ4di4yNTNxMCAuMTI4LS4wMDIuMjUzYy0uMTQgOC4yODQtNi45NyAxNC44ODUtMTUuMjU0IDE0Ljc0NGMtOC4yODQtLjE0LTE0Ljg4NS02Ljk3LTE0Ljc0NC0xNS4yNTNjLjEzOC04LjExNiA2LjY5NC0xNC42MTYgMTQuNzQ5LTE0Ljc0NSIvPjwvc3ZnPg%3D%3D" alt="Segment: change the host">
|
|
27
|
+
<img src="https://img.shields.io/badge/Sentry-change%20the%20DSN-362D59?style=flat-square&labelColor=1f2327&logo=sentry&logoColor=white" alt="Sentry: change the DSN">
|
|
28
|
+
<img src="https://img.shields.io/badge/Python-via%20Segment%20or%20Sentry-3776AB?style=flat-square&labelColor=1f2327&logo=python&logoColor=white" alt="Python: via Segment or Sentry"><br>
|
|
29
|
+
<b>Coming</b> <img src="https://img.shields.io/badge/Rust-coming%20soon-9CA3AF?style=flat-square&labelColor=1f2327&logo=rust&logoColor=white" alt="Rust: coming soon">
|
|
30
|
+
<img src="https://img.shields.io/badge/Kotlin-coming%20soon-9CA3AF?style=flat-square&labelColor=1f2327&logo=kotlin&logoColor=white" alt="Kotlin: coming soon">
|
|
31
|
+
</p>
|
|
32
|
+
<p align="center"><sub>Logos are trademarks of their owners, used only to show compatibility.</sub></p>
|
|
33
|
+
<!-- sdk-badges:end -->
|
|
34
|
+
|
|
35
|
+
Bring your events home. Capture application events into **your own Cloudflare account**, sync them into a local DuckDB file, and give your coding agent raw data for product analysis and reports.
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
your app ──► Worker + Durable Objects ──► R2 archives (in your Cloudflare account)
|
|
39
|
+
│
|
|
40
|
+
slurp sync ◄────┘ (verified, idempotent)
|
|
41
|
+
│
|
|
42
|
+
▼
|
|
43
|
+
DuckDB on your machine ──► SQL, reports, your coding agent
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
No servers to run, no data leaving your account. The Cloudflare Free plan is enough to start.
|
|
47
|
+
|
|
48
|
+
## From user signal to shipped fix
|
|
49
|
+
|
|
50
|
+
<p align="center">
|
|
51
|
+
<img src="docs/assets/slurp-workbench.png" width="900" alt="Diagram: connections send events to Cloudflare R2; your coding agent syncs them into a local DuckDB file, runs SQL analysis, generates a report, ships a product fix and updates the tracking, which collects new signals.">
|
|
52
|
+
</p>
|
|
53
|
+
|
|
54
|
+
Your coding agent owns the whole span: it pulls the data, analyzes it with SQL, writes the report, ships the fix and updates the tracking so the next round of signals arrives. Today the JavaScript SDK and HTTP capture work; the other inputs in the picture (bug reports, chat, crashes, analytics tools, more SDKs) are on the roadmap, see [Connections](https://slurp.run/connections/).
|
|
55
|
+
|
|
56
|
+
## What it costs
|
|
57
|
+
|
|
58
|
+
<!-- pricing:start (generated by apps/site/scripts/readme-pricing.mjs) -->
|
|
59
|
+
| Monthly USD, by events per month | 1M | 10M | 50M | 100M | 1B | 10B |
|
|
60
|
+
|---|--:|--:|--:|--:|--:|--:|
|
|
61
|
+
| [**Slurp, self-hosted on Cloudflare**](docs/self-hosting.md#costs-and-limits) | **Free** | **$0.43** | **$7.96** | **$11.21** | **$42 to $280** | **$420 to $3,200** |
|
|
62
|
+
| [PostHog · Product Analytics](https://posthog.com/docs/product-analytics/pricing) | Free | $324.40 | $1,528.40 | $2,618.40 | $11,618.40 | $92,618.40 |
|
|
63
|
+
| [Mixpanel · Growth (yearly billing)](https://mixpanel.com/pricing/) | Free | $840 | Contact sales | Contact sales | Contact sales | Contact sales |
|
|
64
|
+
| [Amplitude · Plus](https://amplitude.com/pricing) | Free | Not published | Not published | Not published | Not published | Not published |
|
|
65
|
+
| [OpenPanel · Cloud (monthly)](https://openpanel.dev/pricing) | $90 | $350 | $900 | Custom plan | Custom plan | Custom plan |
|
|
66
|
+
|
|
67
|
+
*Vendor prices read from their public pricing pages on November 6, 2026; they change, so check before deciding. Mixpanel is yearly billing. Amplitude does not publish paid event prices, and Mixpanel and OpenPanel direct high volumes to sales.*
|
|
68
|
+
|
|
69
|
+
The Slurp row is the Cloudflare bill for running your own Worker, not a Slurp fee: 20 events per request, about 320 bytes stored per event, 12 months of archives kept, at Cloudflare's list prices (the Workers Free plan covers you until about 50K requests a day). The 1B and 10B figures are ranges, extrapolated from load tests of about 5,000 events a second per source, not months of running those volumes; about two thirds of the bill is R2 storage, so event size and retention matter most. Details: [docs/self-hosting.md](docs/self-hosting.md#costs-and-limits).
|
|
70
|
+
<!-- pricing:end -->
|
|
71
|
+
|
|
72
|
+
## Quickstart
|
|
73
|
+
|
|
74
|
+
[](https://deploy.workers.cloudflare.com/?url=https%3A%2F%2Fgithub.com%2FhitSlop%2Fslurp%2Ftree%2Fmaster%2Fbackend)
|
|
75
|
+
|
|
76
|
+
One click deploys the Worker, Durable Objects and R2 bucket to your Cloudflare account (you choose an `ADMIN_TOKEN`). Then open the Worker's address: it shows the command that connects the CLI. Or do it all from a terminal (to build the CLI from source instead, see [Development](#development)):
|
|
77
|
+
|
|
78
|
+
```sh
|
|
79
|
+
# 1. Install the CLI (macOS or Linux), then deploy your Worker (needs a Cloudflare account and Node 20+)
|
|
80
|
+
curl -LsSf https://slurp.run/install | sh
|
|
81
|
+
# or: npm install -g @hitslop/slurp
|
|
82
|
+
slurp cloud init # logs in, creates the bucket, deploys the Worker, connects this computer
|
|
83
|
+
slurp setup # once per computer: library folder and DuckDB
|
|
84
|
+
|
|
85
|
+
# 2. In your app (npm install @hitslop/slurp-sdk to capture from code, or send from the CLI)
|
|
86
|
+
cd /path/to/your/app
|
|
87
|
+
slurp init --name "My app" # connects this app, opens its quickstart page with your keys
|
|
88
|
+
slurp send signed_up --user-id demo_alice --data '{"demo":true}'
|
|
89
|
+
slurp sync # archives are sealed about every 10 minutes
|
|
90
|
+
slurp sql 'SELECT event, count(*) AS n FROM events GROUP BY event'
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Each project is one event stream on your Worker (many senders can write to it: a website, an app, a webhook) bound to one DuckDB file, stored locally or on an external drive. SQL, report definitions and project metadata stay with your app. No daemon is required. DuckDB 2.x is used for analysis.
|
|
94
|
+
|
|
95
|
+
## Send events
|
|
96
|
+
|
|
97
|
+
Every way lands in the same place: your Worker accepts the event durably, and `slurp sync` can read it after the next archive seal (about 10 minutes by default). You do not make the endpoint or the key by hand: `slurp init` creates your project on the Worker together with its first publishable key, and prints your ingest URL and that key. Those two values are all an SDK needs. A publishable key can only send events, so it is safe in a web page or an app. `slurp project setup PROJECT` shows them again, and `slurp project key create PROJECT --scope publishable --name ios` makes one key per sender. [Where they come from](docs/sdks.md#where-the-endpoint-and-key-come-from).
|
|
98
|
+
|
|
99
|
+
| Send from | Install | Status |
|
|
100
|
+
|---|---|---|
|
|
101
|
+
| **JavaScript / TypeScript** (browser, Node) | `npm install @hitslop/slurp-sdk` | Published |
|
|
102
|
+
| **React Native / Expo** | `npm install @hitslop/slurp-react-native @react-native-async-storage/async-storage` | Tested against fakes; not yet run on a device |
|
|
103
|
+
| **Swift** (iOS, macOS, tvOS, watchOS) | Xcode: add the package `https://github.com/hitSlop/slurp` | Unit tested and run against a local Worker; not yet in an app on a device |
|
|
104
|
+
| **Segment SDKs** you already use | Change the host to your Worker (below) | Verified with analytics-node, analytics-next, analytics-swift and `segment-analytics-python` |
|
|
105
|
+
| **Sentry SDKs** you already use | Change the DSN to your Worker (below) | Verified with `@sentry/node`, `@sentry/browser` and `sentry-sdk` for Python |
|
|
106
|
+
| **Python** | `pip install segment-analytics-python` or `sentry-sdk`, then change the host or DSN | Both run unmodified against a Worker on Python 3.9, 3.12 and 3.14; there is no native Python SDK |
|
|
107
|
+
| **HTTP**, any language | `POST /v1/events` | Tested |
|
|
108
|
+
| **The CLI** | `slurp send`, `slurp identify` | Tested |
|
|
109
|
+
|
|
110
|
+
Coming, with no dates: first-party **Rust** and **Kotlin (Android)** SDKs, and local symbolication of Sentry stack traces in the CLI. Other Segment and Sentry SDKs (Kotlin, Go, Cocoa, Android...) speak the same protocols and should work with the same URL change, but are untested, so they are not claimed. Full recipes and limits: [docs/sdks.md](docs/sdks.md).
|
|
111
|
+
|
|
112
|
+
**JavaScript SDK** (browser and Node, with a durable queue, batching and retries):
|
|
113
|
+
|
|
114
|
+
```js
|
|
115
|
+
import { init, track, identify, flush } from '@hitslop/slurp-sdk';
|
|
116
|
+
|
|
117
|
+
init({
|
|
118
|
+
endpoint: 'https://slurp.YOUR-SUBDOMAIN.workers.dev/v1/events',
|
|
119
|
+
key: 'slurp_pk_…',
|
|
120
|
+
});
|
|
121
|
+
|
|
122
|
+
identify('user_42', { plan: 'pro' });
|
|
123
|
+
track('signed_up', { plan: 'pro' });
|
|
124
|
+
track('feedback', { message: 'Exporting a 4K video fails at 90%', rating: 2 }); // feedback is just another event (or mount the drop-in widget: @hitslop/slurp-sdk/feedback)
|
|
125
|
+
await flush(); // in scripts, send queued events before exiting
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
**React Native / Expo**: events are kept in AsyncStorage, so they survive the app being killed, and are sent when the app goes to the background:
|
|
129
|
+
|
|
130
|
+
```ts
|
|
131
|
+
import { init, track, screen } from '@hitslop/slurp-react-native';
|
|
132
|
+
|
|
133
|
+
init({ endpoint: 'https://slurp.YOUR-SUBDOMAIN.workers.dev/v1/events', key: 'slurp_pk_…' });
|
|
134
|
+
screen('Home');
|
|
135
|
+
track('purchase_completed', { total: 49 });
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
**Swift**: events are written to disk before they are sent:
|
|
139
|
+
|
|
140
|
+
```swift
|
|
141
|
+
import Slurp
|
|
142
|
+
|
|
143
|
+
SlurpClient.configure(.init(key: "slurp_pk_…", endpoint: URL(string: "https://slurp.YOUR-SUBDOMAIN.workers.dev/v1/events")!))
|
|
144
|
+
SlurpClient.shared.screen("Home")
|
|
145
|
+
SlurpClient.shared.track("purchase_completed", ["total": 49])
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
**GitHub, Stripe, RevenueCat or any webhook?** `slurp project webhook PROJECT github` prints a URL and a secret to paste into the provider. Releases, deployments, payments and app purchases then arrive as events (named for the sender, stored whole), a `releases` view lists GitHub releases and deployments so a metric change can be lined up with the release behind it, and a `revenue` view lists RevenueCat purchases and renewals. Preview: [docs/webhooks.md](docs/webhooks.md).
|
|
149
|
+
|
|
150
|
+
**Already using Segment?** Keep the SDK and point it at your Worker. Your publishable key is the write key; `page`, `screen`, `group` and `alias` are stored as `track` events with those names:
|
|
151
|
+
|
|
152
|
+
```js
|
|
153
|
+
// Node (@segment/analytics-node)
|
|
154
|
+
const analytics = new Analytics({ writeKey: 'slurp_pk_…', host: 'https://slurp.YOUR-SUBDOMAIN.workers.dev', path: '/v1/batch' });
|
|
155
|
+
// Browser (@segment/analytics-next)
|
|
156
|
+
const [analytics] = await AnalyticsBrowser.load({ writeKey: 'slurp_pk_…', cdnURL: 'https://slurp.YOUR-SUBDOMAIN.workers.dev' });
|
|
157
|
+
// Swift (analytics-swift)
|
|
158
|
+
let config = Configuration(writeKey: "slurp_pk_…").apiHost("slurp.YOUR-SUBDOMAIN.workers.dev/v1").cdnHost("slurp.YOUR-SUBDOMAIN.workers.dev/v1")
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
**Already using Sentry?** Change the DSN. Errors are stored whole as `error` events and read with the `errors` view; there is no symbolication or issue UI. The DSN's public key is your publishable key as is (the setup page shows the finished DSN):
|
|
162
|
+
|
|
163
|
+
```js
|
|
164
|
+
Sentry.init({ dsn: 'https://slurp_pk_…@slurp.YOUR-SUBDOMAIN.workers.dev/1' });
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
```sql
|
|
168
|
+
SELECT group_key, any_value(exception_type) AS type, count(*) AS n FROM errors GROUP BY group_key ORDER BY n DESC;
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
**The CLI**, from your app's folder, with the project's saved key (handy for testing, scripts and cron jobs):
|
|
172
|
+
|
|
173
|
+
```sh
|
|
174
|
+
slurp send signed_up --user-id user_42 --data '{"plan":"pro"}'
|
|
175
|
+
slurp identify user_42 --data '{"plan":"pro"}'
|
|
176
|
+
slurp send --stdin < events.ndjson # NDJSON records from a file or a pipeline
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
**HTTP from any language**: POST newline-delimited JSON to `/v1/events`:
|
|
180
|
+
|
|
181
|
+
```sh
|
|
182
|
+
curl -X POST "https://slurp.YOUR-SUBDOMAIN.workers.dev/v1/events" \
|
|
183
|
+
-H "Authorization: Bearer slurp_pk_…" \
|
|
184
|
+
-H "Content-Type: application/x-ndjson" \
|
|
185
|
+
--data-binary $'{"id":"evt_1","type":"track","event":"signed_up","timestamp":"2026-11-07T12:00:00Z","user_id":"user_42","data":{"plan":"pro"}}\n'
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
Retrying the exact same bytes is safe: the Worker answers `Slurp-Duplicate: 1`. The event format is described in [docs/architecture.md](docs/architecture.md#contracts).
|
|
189
|
+
|
|
190
|
+
## Connect your agent
|
|
191
|
+
|
|
192
|
+
```sh
|
|
193
|
+
slurp agent setup --out .agents/skills/slurp # skills for any coding agent that runs commands
|
|
194
|
+
claude mcp add slurp -- slurp mcp --project "$PWD" # or MCP: Claude Code, Cursor, Codex, Claude Desktop, Windsurf
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
`slurp mcp` lets an AI client describe the schema, run read-only SQL (in a sandbox that cannot read local files), sync and build reports; `--allow-send` also lets it send events, tagged `context.source = "mcp"`. To post results to your team, `slurp notify slack --channel C0123ABCD --file report/report.png --text "Weekly report"` (Slack bot token) or `slurp notify discord ...` (webhook), run from cron for a scheduled report. Tools, config for each client and a schedule recipe: [docs/integrations.md](docs/integrations.md).
|
|
198
|
+
|
|
199
|
+
## Community
|
|
200
|
+
|
|
201
|
+
Questions, ideas and show-and-tell: [Discord](https://discord.gg/jprRVWSRdn). Bugs and feature requests: [GitHub issues](https://github.com/hitSlop/slurp/issues). The site is [slurp.run](https://slurp.run).
|
|
202
|
+
|
|
203
|
+
## Repository
|
|
204
|
+
|
|
205
|
+
| Path | What |
|
|
206
|
+
|---|---|
|
|
207
|
+
| `backend/` | The Worker (TypeScript): ingest, archives, control plane. Self-contained so it can be deployed on its own. |
|
|
208
|
+
| `crates/` | Rust CLI (`slurp-cli`), client and archive reader (`slurp-client`), Markdown reports (`slurp-document`, `slurp-render`) |
|
|
209
|
+
| `sdks/` | SDKs: [`js`](sdks/js), [`react-native`](sdks/react-native), [`swift`](sdks/swift) (the root `Package.swift` exposes it to Swift Package Manager) |
|
|
210
|
+
| `apps/site` | The public site (home, docs, self-host page, cost comparison) |
|
|
211
|
+
| `apps/setup` | The private offline setup page `slurp init` opens (built into the CLI as `setup.html`) |
|
|
212
|
+
| `docs/` | [Documentation](docs/README.md) |
|
|
213
|
+
|
|
214
|
+
## Development
|
|
215
|
+
|
|
216
|
+
```sh
|
|
217
|
+
pnpm install # website, landing, SDK
|
|
218
|
+
pnpm --filter slurp-setup build # the quickstart page the CLI embeds (crates/slurp-cli/assets/setup.html)
|
|
219
|
+
pnpm --filter slurp-site build
|
|
220
|
+
cargo test --project
|
|
221
|
+
(cd backend && npm install && npm test) # Worker tests under wrangler dev, incl. the real CLI
|
|
222
|
+
(cd sdks/js && node --test test/*.test.ts)
|
|
223
|
+
backend/api/generate.sh --check # regenerate webrpc clients (needs Go; webrpc-gen is pinned in the script)
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
The CLI contract test needs `cargo build -p slurp-cli` and a DuckDB 2 executable (`SLURP_DUCKDB`, default `~/.slurp/bin/duckdb`); it skips cleanly without them.
|
|
227
|
+
|
|
228
|
+
## License
|
|
229
|
+
|
|
230
|
+
MIT. See [LICENSE](LICENSE).
|
|
@@ -0,0 +1,348 @@
|
|
|
1
|
+
const {
|
|
2
|
+
createWriteStream,
|
|
3
|
+
existsSync,
|
|
4
|
+
mkdirSync,
|
|
5
|
+
mkdtemp,
|
|
6
|
+
rmSync,
|
|
7
|
+
} = require("fs");
|
|
8
|
+
const { join, sep } = require("path");
|
|
9
|
+
const { spawnSync } = require("child_process");
|
|
10
|
+
const { tmpdir } = require("os");
|
|
11
|
+
|
|
12
|
+
const https = require("node:https");
|
|
13
|
+
const http = require("node:http");
|
|
14
|
+
|
|
15
|
+
const tmpDir = tmpdir();
|
|
16
|
+
|
|
17
|
+
const error = (msg) => {
|
|
18
|
+
console.error(msg);
|
|
19
|
+
process.exit(1);
|
|
20
|
+
};
|
|
21
|
+
|
|
22
|
+
function getProxyForUrl(urlString) {
|
|
23
|
+
const url = new URL(urlString);
|
|
24
|
+
const isHttps = url.protocol === "https:";
|
|
25
|
+
|
|
26
|
+
const noProxy = process.env.NO_PROXY || process.env.no_proxy || "";
|
|
27
|
+
if (noProxy === "*") return null;
|
|
28
|
+
if (noProxy) {
|
|
29
|
+
const hostname = url.hostname.toLowerCase();
|
|
30
|
+
const noProxyList = noProxy.split(",").map((s) => s.trim().toLowerCase());
|
|
31
|
+
for (const entry of noProxyList) {
|
|
32
|
+
if (hostname === entry || hostname.endsWith("." + entry)) {
|
|
33
|
+
return null;
|
|
34
|
+
}
|
|
35
|
+
}
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
const proxyEnv = isHttps
|
|
39
|
+
? process.env.HTTPS_PROXY || process.env.https_proxy
|
|
40
|
+
: process.env.HTTP_PROXY || process.env.http_proxy;
|
|
41
|
+
|
|
42
|
+
if (!proxyEnv) return null;
|
|
43
|
+
|
|
44
|
+
const proxyUrl = new URL(proxyEnv);
|
|
45
|
+
|
|
46
|
+
let auth = null;
|
|
47
|
+
if (proxyUrl.username || proxyUrl.password) {
|
|
48
|
+
auth = `${proxyUrl.username}:${proxyUrl.password}`;
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
return {
|
|
52
|
+
hostname: proxyUrl.hostname,
|
|
53
|
+
port: proxyUrl.port || (proxyUrl.protocol === "https:" ? 443 : 80),
|
|
54
|
+
auth: auth,
|
|
55
|
+
};
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
function connectThroughProxy(proxy, target) {
|
|
59
|
+
return new Promise((resolve, reject) => {
|
|
60
|
+
const headers = {};
|
|
61
|
+
if (proxy.auth) {
|
|
62
|
+
headers["Proxy-Authorization"] =
|
|
63
|
+
"Basic " + Buffer.from(proxy.auth).toString("base64");
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
const connectReq = http.request({
|
|
67
|
+
hostname: proxy.hostname,
|
|
68
|
+
port: proxy.port,
|
|
69
|
+
method: "CONNECT",
|
|
70
|
+
path: `${target.hostname}:${target.port || 443}`,
|
|
71
|
+
headers,
|
|
72
|
+
});
|
|
73
|
+
connectReq.on("connect", (res, socket) => {
|
|
74
|
+
if (res.statusCode === 200) {
|
|
75
|
+
resolve(socket);
|
|
76
|
+
} else {
|
|
77
|
+
reject(new Error(`Proxy CONNECT failed with status ${res.statusCode}`));
|
|
78
|
+
}
|
|
79
|
+
});
|
|
80
|
+
connectReq.on("error", reject);
|
|
81
|
+
connectReq.end();
|
|
82
|
+
});
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
function download(urlString, maxRedirects) {
|
|
86
|
+
if (maxRedirects === undefined) maxRedirects = 5;
|
|
87
|
+
return new Promise((resolve, reject) => {
|
|
88
|
+
if (maxRedirects < 0) {
|
|
89
|
+
return reject(new Error("Too many redirects"));
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
const parsed = new URL(urlString);
|
|
93
|
+
const isHttps = parsed.protocol === "https:";
|
|
94
|
+
const mod = isHttps ? https : http;
|
|
95
|
+
const proxy = getProxyForUrl(urlString);
|
|
96
|
+
|
|
97
|
+
const doRequest = (extraOptions) => {
|
|
98
|
+
const options = Object.assign(
|
|
99
|
+
{
|
|
100
|
+
hostname: parsed.hostname,
|
|
101
|
+
port: parsed.port || (isHttps ? 443 : 80),
|
|
102
|
+
path: parsed.pathname + parsed.search,
|
|
103
|
+
method: "GET",
|
|
104
|
+
headers: { "User-Agent": "cargo-dist-npm-installer" },
|
|
105
|
+
},
|
|
106
|
+
extraOptions || {},
|
|
107
|
+
);
|
|
108
|
+
|
|
109
|
+
if (proxy && !isHttps) {
|
|
110
|
+
// HTTP through HTTP proxy: request the full URL via the proxy
|
|
111
|
+
options.hostname = proxy.hostname;
|
|
112
|
+
options.port = proxy.port;
|
|
113
|
+
options.path = urlString;
|
|
114
|
+
if (proxy.auth) {
|
|
115
|
+
options.headers["Proxy-Authorization"] =
|
|
116
|
+
"Basic " + Buffer.from(proxy.auth).toString("base64");
|
|
117
|
+
}
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
const req = mod.request(options, (res) => {
|
|
121
|
+
if (
|
|
122
|
+
res.statusCode >= 300 &&
|
|
123
|
+
res.statusCode < 400 &&
|
|
124
|
+
res.headers.location
|
|
125
|
+
) {
|
|
126
|
+
res.resume();
|
|
127
|
+
const nextUrl = new URL(res.headers.location, urlString).toString();
|
|
128
|
+
return download(nextUrl, maxRedirects - 1).then(resolve, reject);
|
|
129
|
+
}
|
|
130
|
+
if (res.statusCode < 200 || res.statusCode >= 300) {
|
|
131
|
+
res.resume();
|
|
132
|
+
return reject(new Error(`HTTP ${res.statusCode} from ${urlString}`));
|
|
133
|
+
}
|
|
134
|
+
resolve(res);
|
|
135
|
+
});
|
|
136
|
+
req.on("error", reject);
|
|
137
|
+
req.end();
|
|
138
|
+
};
|
|
139
|
+
|
|
140
|
+
if (proxy && isHttps) {
|
|
141
|
+
connectThroughProxy(proxy, parsed).then(
|
|
142
|
+
(socket) => doRequest({ socket, agent: false }),
|
|
143
|
+
reject,
|
|
144
|
+
);
|
|
145
|
+
} else {
|
|
146
|
+
doRequest();
|
|
147
|
+
}
|
|
148
|
+
});
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
class Package {
|
|
152
|
+
constructor(platform, name, url, filename, zipExt, binaries) {
|
|
153
|
+
let errors = [];
|
|
154
|
+
if (typeof url !== "string") {
|
|
155
|
+
errors.push("url must be a string");
|
|
156
|
+
} else {
|
|
157
|
+
try {
|
|
158
|
+
new URL(url);
|
|
159
|
+
} catch (e) {
|
|
160
|
+
errors.push(e);
|
|
161
|
+
}
|
|
162
|
+
}
|
|
163
|
+
if (name && typeof name !== "string") {
|
|
164
|
+
errors.push("package name must be a string");
|
|
165
|
+
}
|
|
166
|
+
if (!name) {
|
|
167
|
+
errors.push("You must specify the name of your package");
|
|
168
|
+
}
|
|
169
|
+
if (binaries && typeof binaries !== "object") {
|
|
170
|
+
errors.push("binaries must be a string => string map");
|
|
171
|
+
}
|
|
172
|
+
if (!binaries) {
|
|
173
|
+
errors.push("You must specify the binaries in the package");
|
|
174
|
+
}
|
|
175
|
+
|
|
176
|
+
if (errors.length > 0) {
|
|
177
|
+
let errorMsg =
|
|
178
|
+
"One or more of the parameters you passed to the Binary constructor are invalid:\n";
|
|
179
|
+
errors.forEach((error) => {
|
|
180
|
+
errorMsg += error;
|
|
181
|
+
});
|
|
182
|
+
errorMsg +=
|
|
183
|
+
'\n\nCorrect usage: new Package("my-binary", "https://example.com/binary/download.tar.gz", {"my-binary": "my-binary"})';
|
|
184
|
+
error(errorMsg);
|
|
185
|
+
}
|
|
186
|
+
|
|
187
|
+
this.platform = platform;
|
|
188
|
+
this.url = url;
|
|
189
|
+
this.name = name;
|
|
190
|
+
this.filename = filename;
|
|
191
|
+
this.zipExt = zipExt;
|
|
192
|
+
this.installDirectory = join(__dirname, "node_modules", ".bin_real");
|
|
193
|
+
this.binaries = binaries;
|
|
194
|
+
|
|
195
|
+
if (!existsSync(this.installDirectory)) {
|
|
196
|
+
mkdirSync(this.installDirectory, { recursive: true });
|
|
197
|
+
}
|
|
198
|
+
}
|
|
199
|
+
|
|
200
|
+
exists() {
|
|
201
|
+
for (const binaryName in this.binaries) {
|
|
202
|
+
const binRelPath = this.binaries[binaryName];
|
|
203
|
+
const binPath = join(this.installDirectory, binRelPath);
|
|
204
|
+
if (!existsSync(binPath)) {
|
|
205
|
+
return false;
|
|
206
|
+
}
|
|
207
|
+
}
|
|
208
|
+
return true;
|
|
209
|
+
}
|
|
210
|
+
|
|
211
|
+
install(suppressLogs = false) {
|
|
212
|
+
if (this.exists()) {
|
|
213
|
+
if (!suppressLogs) {
|
|
214
|
+
console.error(
|
|
215
|
+
`${this.name} is already installed, skipping installation.`,
|
|
216
|
+
);
|
|
217
|
+
}
|
|
218
|
+
return Promise.resolve();
|
|
219
|
+
}
|
|
220
|
+
|
|
221
|
+
try {
|
|
222
|
+
rmSync(this.installDirectory, { recursive: true, force: true });
|
|
223
|
+
} catch {
|
|
224
|
+
// ignore - directory may not exist
|
|
225
|
+
}
|
|
226
|
+
|
|
227
|
+
mkdirSync(this.installDirectory, { recursive: true });
|
|
228
|
+
|
|
229
|
+
if (!suppressLogs) {
|
|
230
|
+
console.error(`Downloading release from ${this.url}`);
|
|
231
|
+
}
|
|
232
|
+
|
|
233
|
+
return download(this.url)
|
|
234
|
+
.then((res) => {
|
|
235
|
+
return new Promise((resolve, reject) => {
|
|
236
|
+
mkdtemp(`${tmpDir}${sep}`, (err, directory) => {
|
|
237
|
+
if (err) return reject(err);
|
|
238
|
+
let tempFile = join(directory, this.filename);
|
|
239
|
+
const sink = res.pipe(createWriteStream(tempFile));
|
|
240
|
+
sink.on("error", (err) => reject(err));
|
|
241
|
+
sink.on("close", () => {
|
|
242
|
+
if (/\.tar\.*/.test(this.zipExt)) {
|
|
243
|
+
const result = spawnSync("tar", [
|
|
244
|
+
"xf",
|
|
245
|
+
tempFile,
|
|
246
|
+
// The tarballs are stored with a leading directory
|
|
247
|
+
// component; we strip one component in the
|
|
248
|
+
// shell installers too.
|
|
249
|
+
"--strip-components",
|
|
250
|
+
"1",
|
|
251
|
+
"-C",
|
|
252
|
+
this.installDirectory,
|
|
253
|
+
]);
|
|
254
|
+
if (result.status == 0) {
|
|
255
|
+
resolve();
|
|
256
|
+
} else if (result.error) {
|
|
257
|
+
reject(result.error);
|
|
258
|
+
} else {
|
|
259
|
+
reject(
|
|
260
|
+
new Error(
|
|
261
|
+
`An error occurred untarring the artifact: stdout: ${result.stdout}; stderr: ${result.stderr}`,
|
|
262
|
+
),
|
|
263
|
+
);
|
|
264
|
+
}
|
|
265
|
+
} else if (this.zipExt == ".zip") {
|
|
266
|
+
let result;
|
|
267
|
+
if (this.platform.artifactName.includes("windows")) {
|
|
268
|
+
// Windows does not have "unzip" by default on many installations, instead
|
|
269
|
+
// we use Expand-Archive from powershell
|
|
270
|
+
result = spawnSync("powershell.exe", [
|
|
271
|
+
"-NoProfile",
|
|
272
|
+
"-NonInteractive",
|
|
273
|
+
"-Command",
|
|
274
|
+
`& {
|
|
275
|
+
param([string]$LiteralPath, [string]$DestinationPath)
|
|
276
|
+
Expand-Archive -LiteralPath $LiteralPath -DestinationPath $DestinationPath -Force
|
|
277
|
+
}`,
|
|
278
|
+
tempFile,
|
|
279
|
+
this.installDirectory,
|
|
280
|
+
]);
|
|
281
|
+
} else {
|
|
282
|
+
result = spawnSync("unzip", [
|
|
283
|
+
"-q",
|
|
284
|
+
tempFile,
|
|
285
|
+
"-d",
|
|
286
|
+
this.installDirectory,
|
|
287
|
+
]);
|
|
288
|
+
}
|
|
289
|
+
|
|
290
|
+
if (result.status == 0) {
|
|
291
|
+
resolve();
|
|
292
|
+
} else if (result.error) {
|
|
293
|
+
reject(result.error);
|
|
294
|
+
} else {
|
|
295
|
+
reject(
|
|
296
|
+
new Error(
|
|
297
|
+
`An error occurred unzipping the artifact: stdout: ${result.stdout}; stderr: ${result.stderr}`,
|
|
298
|
+
),
|
|
299
|
+
);
|
|
300
|
+
}
|
|
301
|
+
} else {
|
|
302
|
+
reject(
|
|
303
|
+
new Error(`Unrecognized file extension: ${this.zipExt}`),
|
|
304
|
+
);
|
|
305
|
+
}
|
|
306
|
+
});
|
|
307
|
+
});
|
|
308
|
+
});
|
|
309
|
+
})
|
|
310
|
+
.then(() => {
|
|
311
|
+
if (!suppressLogs) {
|
|
312
|
+
console.error(`${this.name} has been installed!`);
|
|
313
|
+
}
|
|
314
|
+
})
|
|
315
|
+
.catch((e) => {
|
|
316
|
+
error(`Error fetching release: ${e.message}`);
|
|
317
|
+
});
|
|
318
|
+
}
|
|
319
|
+
|
|
320
|
+
run(binaryName) {
|
|
321
|
+
const promise = !this.exists() ? this.install(true) : Promise.resolve();
|
|
322
|
+
|
|
323
|
+
promise
|
|
324
|
+
.then(() => {
|
|
325
|
+
const [, , ...args] = process.argv;
|
|
326
|
+
|
|
327
|
+
const options = { cwd: process.cwd(), stdio: "inherit" };
|
|
328
|
+
|
|
329
|
+
const binRelPath = this.binaries[binaryName];
|
|
330
|
+
if (!binRelPath) {
|
|
331
|
+
error(`${binaryName} is not a known binary in ${this.name}`);
|
|
332
|
+
}
|
|
333
|
+
const binPath = join(this.installDirectory, binRelPath);
|
|
334
|
+
const result = spawnSync(binPath, args, options);
|
|
335
|
+
|
|
336
|
+
if (result.error) {
|
|
337
|
+
error(result.error);
|
|
338
|
+
}
|
|
339
|
+
|
|
340
|
+
process.exit(result.status);
|
|
341
|
+
})
|
|
342
|
+
.catch((e) => {
|
|
343
|
+
error(e.message);
|
|
344
|
+
});
|
|
345
|
+
}
|
|
346
|
+
}
|
|
347
|
+
|
|
348
|
+
module.exports.Package = Package;
|
package/binary.js
ADDED
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
const { Package } = require("./binary-install");
|
|
2
|
+
const os = require("os");
|
|
3
|
+
const libc = require("detect-libc");
|
|
4
|
+
|
|
5
|
+
const error = (msg) => {
|
|
6
|
+
console.error(msg);
|
|
7
|
+
process.exit(1);
|
|
8
|
+
};
|
|
9
|
+
|
|
10
|
+
const {
|
|
11
|
+
name,
|
|
12
|
+
artifactDownloadUrls,
|
|
13
|
+
supportedPlatforms,
|
|
14
|
+
glibcMinimum,
|
|
15
|
+
} = require("./package.json");
|
|
16
|
+
|
|
17
|
+
// FIXME: implement NPM installer handling of fallback download URLs
|
|
18
|
+
const artifactDownloadUrl = artifactDownloadUrls[0];
|
|
19
|
+
const builderGlibcMajorVersion = glibcMinimum.major;
|
|
20
|
+
const builderGlibcMinorVersion = glibcMinimum.series;
|
|
21
|
+
|
|
22
|
+
const getPlatform = () => {
|
|
23
|
+
const rawOsType = os.type();
|
|
24
|
+
const rawArchitecture = os.arch();
|
|
25
|
+
|
|
26
|
+
// We want to use rust-style target triples as the canonical key
|
|
27
|
+
// for a platform, so translate the "os" library's concepts into rust ones
|
|
28
|
+
let osType = "";
|
|
29
|
+
switch (rawOsType) {
|
|
30
|
+
case "Windows_NT":
|
|
31
|
+
osType = "pc-windows-msvc";
|
|
32
|
+
break;
|
|
33
|
+
case "Darwin":
|
|
34
|
+
osType = "apple-darwin";
|
|
35
|
+
break;
|
|
36
|
+
case "Linux":
|
|
37
|
+
osType = "unknown-linux-gnu";
|
|
38
|
+
break;
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
let arch = "";
|
|
42
|
+
switch (rawArchitecture) {
|
|
43
|
+
case "x64":
|
|
44
|
+
arch = "x86_64";
|
|
45
|
+
break;
|
|
46
|
+
case "arm64":
|
|
47
|
+
arch = "aarch64";
|
|
48
|
+
break;
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
if (rawOsType === "Linux") {
|
|
52
|
+
if (libc.familySync() == "musl") {
|
|
53
|
+
osType = "unknown-linux-musl-dynamic";
|
|
54
|
+
} else if (libc.isNonGlibcLinuxSync()) {
|
|
55
|
+
console.warn(
|
|
56
|
+
"Your libc is neither glibc nor musl; trying static musl binary instead",
|
|
57
|
+
);
|
|
58
|
+
osType = "unknown-linux-musl-static";
|
|
59
|
+
} else {
|
|
60
|
+
let libcVersion = libc.versionSync();
|
|
61
|
+
let splitLibcVersion = libcVersion.split(".");
|
|
62
|
+
let libcMajorVersion = splitLibcVersion[0];
|
|
63
|
+
let libcMinorVersion = splitLibcVersion[1];
|
|
64
|
+
if (
|
|
65
|
+
libcMajorVersion != builderGlibcMajorVersion ||
|
|
66
|
+
libcMinorVersion < builderGlibcMinorVersion
|
|
67
|
+
) {
|
|
68
|
+
// We can't run the glibc binaries, but we can run the static musl ones
|
|
69
|
+
// if they exist
|
|
70
|
+
console.warn(
|
|
71
|
+
"Your glibc isn't compatible; trying static musl binary instead",
|
|
72
|
+
);
|
|
73
|
+
osType = "unknown-linux-musl-static";
|
|
74
|
+
}
|
|
75
|
+
}
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
// Assume the above succeeded and build a target triple to look things up with.
|
|
79
|
+
// If any of it failed, this lookup will fail and we'll handle it like normal.
|
|
80
|
+
let targetTriple = `${arch}-${osType}`;
|
|
81
|
+
let platform = supportedPlatforms[targetTriple];
|
|
82
|
+
|
|
83
|
+
if (!platform) {
|
|
84
|
+
error(
|
|
85
|
+
`Platform with type "${rawOsType}" and architecture "${rawArchitecture}" is not supported by ${name}.\nYour system must be one of the following:\n\n${Object.keys(
|
|
86
|
+
supportedPlatforms,
|
|
87
|
+
).join(",")}`,
|
|
88
|
+
);
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
return platform;
|
|
92
|
+
};
|
|
93
|
+
|
|
94
|
+
const getPackage = () => {
|
|
95
|
+
const platform = getPlatform();
|
|
96
|
+
const url = `${artifactDownloadUrl}/${platform.artifactName}`;
|
|
97
|
+
let filename = platform.artifactName;
|
|
98
|
+
let ext = platform.zipExt;
|
|
99
|
+
let binary = new Package(platform, name, url, filename, ext, platform.bins);
|
|
100
|
+
|
|
101
|
+
return binary;
|
|
102
|
+
};
|
|
103
|
+
|
|
104
|
+
const install = (suppressLogs) => {
|
|
105
|
+
if (!artifactDownloadUrl || artifactDownloadUrl.length === 0) {
|
|
106
|
+
console.warn("in demo mode, not installing binaries");
|
|
107
|
+
return;
|
|
108
|
+
}
|
|
109
|
+
const pkg = getPackage();
|
|
110
|
+
|
|
111
|
+
return pkg.install(suppressLogs);
|
|
112
|
+
};
|
|
113
|
+
|
|
114
|
+
const run = (binaryName) => {
|
|
115
|
+
const pkg = getPackage();
|
|
116
|
+
|
|
117
|
+
pkg.run(binaryName);
|
|
118
|
+
};
|
|
119
|
+
|
|
120
|
+
module.exports = {
|
|
121
|
+
install,
|
|
122
|
+
run,
|
|
123
|
+
getPackage,
|
|
124
|
+
};
|
package/install.js
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
{
|
|
2
|
+
"lockfileVersion": 3,
|
|
3
|
+
"name": "@hitslop/slurp",
|
|
4
|
+
"packages": {
|
|
5
|
+
"": {
|
|
6
|
+
"bin": {
|
|
7
|
+
"slurp": "run-slurp.js"
|
|
8
|
+
},
|
|
9
|
+
"dependencies": {
|
|
10
|
+
"detect-libc": "^2.1.2"
|
|
11
|
+
},
|
|
12
|
+
"devDependencies": {
|
|
13
|
+
"prettier": "^3.9.10"
|
|
14
|
+
},
|
|
15
|
+
"engines": {
|
|
16
|
+
"node": ">=14.18",
|
|
17
|
+
"npm": ">=6"
|
|
18
|
+
},
|
|
19
|
+
"hasInstallScript": true,
|
|
20
|
+
"license": "MIT",
|
|
21
|
+
"name": "@hitslop/slurp",
|
|
22
|
+
"version": "0.1.1"
|
|
23
|
+
},
|
|
24
|
+
"node_modules/detect-libc": {
|
|
25
|
+
"engines": {
|
|
26
|
+
"node": ">=8"
|
|
27
|
+
},
|
|
28
|
+
"integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
|
|
29
|
+
"license": "Apache-2.0",
|
|
30
|
+
"resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
|
|
31
|
+
"version": "2.1.2"
|
|
32
|
+
},
|
|
33
|
+
"node_modules/prettier": {
|
|
34
|
+
"bin": {
|
|
35
|
+
"prettier": "bin/prettier.cjs"
|
|
36
|
+
},
|
|
37
|
+
"dev": true,
|
|
38
|
+
"engines": {
|
|
39
|
+
"node": ">=14"
|
|
40
|
+
},
|
|
41
|
+
"funding": {
|
|
42
|
+
"url": "https://github.com/prettier/prettier?sponsor=1"
|
|
43
|
+
},
|
|
44
|
+
"integrity": "sha512-skGHTH1Yb9QvaTC4hWnRTzEYtCpApF0UMBPz+t/0ZPzP8ihefo/btKd+QA7pUE0fEwA1SbS/NmhyEcstdXOIdA==",
|
|
45
|
+
"license": "MIT",
|
|
46
|
+
"resolved": "https://registry.npmjs.org/prettier/-/prettier-3.9.10.tgz",
|
|
47
|
+
"version": "3.9.10"
|
|
48
|
+
}
|
|
49
|
+
},
|
|
50
|
+
"requires": true,
|
|
51
|
+
"version": "0.1.1"
|
|
52
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,66 @@
|
|
|
1
1
|
{
|
|
2
|
+
"artifactDownloadUrls": [
|
|
3
|
+
"https://github.com/hitSlop/slurp/releases/download/v0.1.1"
|
|
4
|
+
],
|
|
5
|
+
"bin": {
|
|
6
|
+
"slurp": "run-slurp.js"
|
|
7
|
+
},
|
|
8
|
+
"dependencies": {
|
|
9
|
+
"detect-libc": "^2.1.2"
|
|
10
|
+
},
|
|
11
|
+
"devDependencies": {
|
|
12
|
+
"prettier": "^3.9.10"
|
|
13
|
+
},
|
|
14
|
+
"engines": {
|
|
15
|
+
"node": ">=14.18",
|
|
16
|
+
"npm": ">=6"
|
|
17
|
+
},
|
|
18
|
+
"glibcMinimum": {
|
|
19
|
+
"major": 2,
|
|
20
|
+
"series": 35
|
|
21
|
+
},
|
|
22
|
+
"license": "MIT",
|
|
2
23
|
"name": "@hitslop/slurp",
|
|
3
|
-
"
|
|
4
|
-
"
|
|
5
|
-
"
|
|
24
|
+
"preferUnplugged": true,
|
|
25
|
+
"repository": "https://github.com/hitSlop/slurp",
|
|
26
|
+
"scripts": {
|
|
27
|
+
"fmt": "prettier --write **/*.js",
|
|
28
|
+
"fmt:check": "prettier --check **/*.js",
|
|
29
|
+
"postinstall": "node ./install.js"
|
|
30
|
+
},
|
|
31
|
+
"supportedPlatforms": {
|
|
32
|
+
"aarch64-apple-darwin": {
|
|
33
|
+
"artifactName": "slurp-cli-aarch64-apple-darwin.tar.xz",
|
|
34
|
+
"bins": {
|
|
35
|
+
"slurp": "slurp"
|
|
36
|
+
},
|
|
37
|
+
"zipExt": ".tar.xz"
|
|
38
|
+
},
|
|
39
|
+
"aarch64-unknown-linux-gnu": {
|
|
40
|
+
"artifactName": "slurp-cli-aarch64-unknown-linux-gnu.tar.xz",
|
|
41
|
+
"bins": {
|
|
42
|
+
"slurp": "slurp"
|
|
43
|
+
},
|
|
44
|
+
"zipExt": ".tar.xz"
|
|
45
|
+
},
|
|
46
|
+
"x86_64-apple-darwin": {
|
|
47
|
+
"artifactName": "slurp-cli-x86_64-apple-darwin.tar.xz",
|
|
48
|
+
"bins": {
|
|
49
|
+
"slurp": "slurp"
|
|
50
|
+
},
|
|
51
|
+
"zipExt": ".tar.xz"
|
|
52
|
+
},
|
|
53
|
+
"x86_64-unknown-linux-gnu": {
|
|
54
|
+
"artifactName": "slurp-cli-x86_64-unknown-linux-gnu.tar.xz",
|
|
55
|
+
"bins": {
|
|
56
|
+
"slurp": "slurp"
|
|
57
|
+
},
|
|
58
|
+
"zipExt": ".tar.xz"
|
|
59
|
+
}
|
|
60
|
+
},
|
|
61
|
+
"version": "0.1.1",
|
|
62
|
+
"volta": {
|
|
63
|
+
"node": "18.14.1",
|
|
64
|
+
"npm": "9.5.0"
|
|
65
|
+
}
|
|
6
66
|
}
|
package/run-slurp.js
ADDED