sphere-cli 0.1.38 → 0.1.39

package/README.md

@@ -49,6 +49,9 @@ sh install.sh --uninstall
 ## Quick start
 
 ```sh
+# Try the built-in demo (no data needed)
+sphere demo
+
 # Activate your license (once)
 sphere license activate sphere_xxxxxxxxxxxxxxxxxxxx
 
@@ -64,8 +67,75 @@ sphere certify real.csv synth.csv -o report.html
 
 ---
 
+## First run
+
+On the very first invocation the CLI cold-loads its bundled Python libraries (pandas, pyarrow, anonymeter, sklearn) from disk. On Apple Silicon this typically takes **15–25 seconds** and is shown in the progress bar as each library finishes:
+
+```
+Generating synthetic data from nhanes_sample.csv …
+  [░░░░░░░░░░░░░░░░░]   0.0%  loading pandas . .
+  [█░░░░░░░░░░░░░░░░]   3.0%  ✓ pandas  (12.4 s)
+  [██░░░░░░░░░░░░░░░]   6.0%  ✓ pyarrow  (3.1 s)
+  [███░░░░░░░░░░░░░░]   9.0%  ✓ sphere core  (1.8 s)
+  …
+✓ synth.csv  4,899 rows × 18 cols  (load 17.4 s + run 1.8 s)  seed 3721018536
+```
+
+Subsequent calls in the same session skip loading entirely. The timing line always shows **load** (library startup) and **run** (actual SPHERE computation) separately so you can see which part is slow.
+
+> Exact times vary by machine, OS page cache state, and whether the binary has been run recently.
+
+---
+
 ## Commands
 
+### `sphere demo`
+
+Run SPHERE end-to-end on the built-in NHANES sample dataset (4,899 rows × 18 columns, mix of continuous and categorical variables). No data or license required — good for testing an installation.
+
+```sh
+sphere demo
+```
+
+```
+SPHERE demo — built-in NHANES dataset (4,899 rows × 18 cols, continuous + categorical)
+────────────────────────────────────────────────────
+
+Generating synthetic data from nhanes_sample.csv …
+  [░░░░░░░░░░░░░░░░░]   0.0%  loading pandas . .
+  [█░░░░░░░░░░░░░░░░]   3.0%  ✓ pandas  (12.4 s)
+  [██░░░░░░░░░░░░░░░]   6.0%  ✓ pyarrow  (3.1 s)
+  [███░░░░░░░░░░░░░░]   9.0%  ✓ sphere core  (1.8 s)
+  [████████████████░]  85.0%  writing output
+✓ /tmp/synth.csv  4,899 rows × 18 cols  (load 17.4 s + run 1.8 s)  seed 3721018536
+
+Evaluating nhanes_sample.csv vs synth.csv …
+  [████░░░░░░░░░░░░░]  16.0%  loading anonymeter . .
+  [████░░░░░░░░░░░░░]  17.0%  ✓ anonymeter  (3.2 s)
+  [█████░░░░░░░░░░░░]  18.0%  ✓ sklearn  (0.8 s)
+  [█████████████████]  89.0%  inference  9/9
+✓ Evaluation complete  (load 4.0 s + run 14.2 s)
+
+  Fidelity
+  ────────────────────────────────────
+  Mean           100.0  ████████████████████
+  Variance        99.7  ████████████████████
+  Correlation     95.1  ███████████████████░
+  KS              96.8  ███████████████████░
+  ────────────────────────────────────
+  Composite       97.9  ████████████████████
+
+  Privacy
+  ────────────────────────────────────
+  Singling Out   100.0  ████████████████████
+  Linkability     97.5  ███████████████████░
+  Inference       96.8  ███████████████████░
+  ────────────────────────────────────
+  Composite       98.1  ████████████████████
+```
+
+---
+
 ### `sphere license`
 
 Activate and manage your SPHERE license. A valid license is required to use `generate`, `evaluate`, and `certify`.
@@ -90,7 +160,7 @@ sphere generate <real.csv> [options]
 Options:
   -o, --output PATH        Output CSV path (default: <input>_sphere.csv)
   -n, --rows INT           Number of synthetic rows (default: same as input)
-  -k INT                   SPHERE rotations (default: 2)
+  -k INT                   Synthesis depth (default: 2)
   --seed INT               Random seed for reproducibility
   --mix-prob FLOAT         Mixture probability 0–1 (default: 0.75)
   --json                   Machine-readable JSON output
@@ -98,6 +168,8 @@ Options:
 
 A `.sphere.json` provenance file is written alongside every output CSV and is automatically read by `sphere certify`.
 
+---
+
 ### `sphere evaluate`
 
 ```
@@ -105,9 +177,14 @@ sphere evaluate <real.csv> <synth.csv> [options]
 
 Options:
   --skip-privacy           Skip privacy metrics (faster)
+  --seed INT               Fix the random seed for reproducible attack results
   --json                   Machine-readable JSON output
 ```
 
+Reports four fidelity metrics (mean, variance, correlation, KS) and three privacy metrics (singling-out, linkability, inference), each scored 0–100. Scores are normalised against a column-shuffled baseline so 100 = no measurable privacy leakage relative to a random permutation of the data.
+
+---
+
 ### `sphere certify`
 
 ```
@@ -118,7 +195,7 @@ Options:
   --json                   Machine-readable JSON output
 ```
 
-Generation parameters (`k`, `seed`, `theta`, etc.) are loaded automatically from the `.sphere.json` sidecar. Pass flags explicitly to override.
+Produces a self-contained HTML certificate with fidelity and privacy scores, dataset metadata, and generation provenance. Generation parameters (`k`, `seed`, `theta`, etc.) are loaded automatically from the `.sphere.json` sidecar; pass flags explicitly to override.
 
 ---
 
@@ -127,8 +204,9 @@ Generation parameters (`k`, `seed`, `theta`, etc.) are loaded automatically from
 Every command supports `--json` for pipeline integration:
 
 ```sh
-sphere generate real.csv -o synth.csv --json | jq .fidelity
+sphere generate real.csv -o synth.csv --json | jq .seed
 sphere evaluate real.csv synth.csv --json > metrics.json
+sphere evaluate real.csv synth.csv --json | jq '.privacy.composite'
 ```
 
 ---
@@ -140,7 +218,7 @@ sphere evaluate real.csv synth.csv --json > metrics.json
 | `SPHERE_LICENSE_REQUIRED` | Set to `false` to bypass license checks (research / unlocked builds) |
 | `SPHERE_WORKER_URL` | Override the license validation endpoint |
 | `SPHERE_PREFIX` | Override install prefix |
-| `SPHERE_VERSION` | Pin a release tag, e.g. `v0.1.0` |
+| `SPHERE_VERSION` | Pin a release tag, e.g. `v0.1.38` |
 | `SPHERE_BUNDLE_URL` | Full URL to a `sphere-cli-*.tar.gz` (skip auto-detect) |
 | `SPHERE_GITHUB_REPO` | Override GitHub repo for downloads |
 

package/bin/sphere.js

@@ -1,24 +1,444 @@
 #!/usr/bin/env node
 'use strict';
 
-const { spawnSync } = require('child_process');
-const path = require('path');
-const fs = require('fs');
+/**
+ * sphere — CLI entry point.
+ *
+ * This wrapper handles argument parsing, progress display, and license checks,
+ * then delegates generation to sphere-node.jsc (V8 bytecode, bytenode-loaded).
+ *
+ * Commands:
+ *   sphere generate  <input.csv>  -o <output.csv>  [options]
+ *   sphere license   activate|status|clear
+ *   sphere demo
+ *   sphere evaluate  — not available in this version (requires Python sidecar)
+ *   sphere certify   — not available in this version (requires Python sidecar)
+ */
 
-const binaryPath = path.join(__dirname, '..', 'native', 'sphere-cli', 'sphere');
+const { spawn } = require('child_process');
+const path   = require('path');
+const fs     = require('fs');
+const https  = require('https');
+const os     = require('os');
 
-if (!fs.existsSync(binaryPath)) {
-  console.error(
-    'SPHERE CLI binary not found.\n' +
-    'This usually means the postinstall step failed or was skipped.\n' +
-    'Try reinstalling:  npm install -g sphere-cli'
+const PKG_DIR        = path.join(__dirname, '..');
+const SPHERE_NODE_JS = path.join(PKG_DIR, 'sphere-node.js');
+const VERSION        = require(path.join(PKG_DIR, 'package.json')).version;
+
+// ── License ───────────────────────────────────────────────────────────────────
+
+const LICENSE_WORKER_URL = process.env.SPHERE_WORKER_URL
+  || 'https://sphere-license.statzihuai.workers.dev';
+const LICENSE_CACHE_DAYS = 1;
+const LICENSE_REQUIRED   = (process.env.SPHERE_LICENSE_REQUIRED || 'true') !== 'false';
+
+function sphereConfigDir() {
+  const d = path.join(os.homedir(), '.config', 'sphere');
+  fs.mkdirSync(d, { recursive: true });
+  return d;
+}
+function licenseKeyFile()   { return path.join(sphereConfigDir(), 'license_key'); }
+function licenseCacheFile() { return path.join(sphereConfigDir(), 'license_cache.json'); }
+
+function readStoredLicenseKey() {
+  const f = licenseKeyFile();
+  if (!fs.existsSync(f)) return null;
+  return fs.readFileSync(f, 'utf8').trim() || null;
+}
+
+function writeLicenseKey(key) {
+  const f = licenseKeyFile();
+  fs.writeFileSync(f, key, { encoding: 'utf8', mode: 0o600 });
+}
+
+function readLicenseCache() {
+  const f = licenseCacheFile();
+  if (!fs.existsSync(f)) return null;
+  try {
+    const data = JSON.parse(fs.readFileSync(f, 'utf8'));
+    const ageDays = (Date.now() / 1000 - (data.cachedAt || 0)) / 86400;
+    return ageDays <= LICENSE_CACHE_DAYS ? data : null;
+  } catch { return null; }
+}
+
+function writeLicenseCache(data) {
+  fs.writeFileSync(
+    licenseCacheFile(),
+    JSON.stringify({ ...data, cachedAt: Math.floor(Date.now() / 1000) }),
+    'utf8',
   );
+}
+
+function validateKeyOnline(key) {
+  return new Promise((resolve, reject) => {
+    const body = Buffer.from(JSON.stringify({ key }));
+    const req = https.request(
+      `${LICENSE_WORKER_URL}/validate`,
+      {
+        method: 'POST',
+        headers: {
+          'Content-Type': 'application/json',
+          'Content-Length': body.length,
+          'User-Agent': 'sphere-cli/1.0',
+        },
+      },
+      (res) => {
+        let buf = '';
+        res.on('data', (d) => (buf += d));
+        res.on('end', () => {
+          try { resolve(JSON.parse(buf)); }
+          catch { reject(new Error('Invalid JSON from license server')); }
+        });
+      },
+    );
+    req.on('error', reject);
+    req.setTimeout(10000, () => { req.destroy(new Error('License server timeout')); });
+    req.write(body);
+    req.end();
+  });
+}
+
+async function checkLicense() {
+  if (!LICENSE_REQUIRED) return;
+  const key = readStoredLicenseKey();
+  if (!key) {
+    process.stderr.write(
+      '✗  No SPHERE license found.\n' +
+      '   Run:  sphere license activate <key>\n' +
+      '   Contact zihuai@stanford.edu to get a license.\n',
+    );
+    process.exit(1);
+  }
+
+  let result;
+  try {
+    result = await validateKeyOnline(key);
+    writeLicenseCache(result);
+  } catch {
+    result = readLicenseCache();
+    if (!result) {
+      process.stderr.write(
+        '✗  License server unreachable and local cache has expired.\n' +
+        '   Connect to the internet and re-run to refresh your license.\n',
+      );
+      process.exit(1);
+    }
+  }
+
+  if (!result.valid) {
+    process.stderr.write(
+      `✗  License invalid: ${result.error || 'unknown error'}\n` +
+      '   Run:  sphere license activate <key>\n',
+    );
+    process.exit(1);
+  }
+}
+
+// ── Progress bar ──────────────────────────────────────────────────────────────
+
+const BAR_WIDTH = 28;
+
+function progressBar(frac, msg) {
+  const f = Math.min(1, Math.max(0, frac));
+  const filled = Math.round(BAR_WIDTH * f);
+  const bar = '█'.repeat(filled) + '░'.repeat(BAR_WIDTH - filled);
+  const pct = (f * 100).toFixed(1).padStart(5);
+  process.stderr.write(`\r  [${bar}] ${pct}%  ${(msg || '').slice(0, 45).padEnd(45)}\x1b[K`);
+}
+
+function clearLine() {
+  process.stderr.write(`\r${' '.repeat(BAR_WIDTH + 60)}\r`);
+}
+
+// ── Argument parsing helpers ──────────────────────────────────────────────────
+
+function parseArgs(argv) {
+  const args = argv.slice(2);
+  const pos = [];
+  const flags = {};
+  for (let i = 0; i < args.length; i++) {
+    const a = args[i];
+    if (a === '-o') {
+      flags['output'] = args[++i];
+    } else if (a.startsWith('--')) {
+      const key = a.slice(2);
+      const next = args[i + 1];
+      if (next === undefined || next.startsWith('-')) {
+        flags[key] = true;
+      } else {
+        flags[key] = next;
+        i++;
+      }
+    } else if (a.startsWith('-') && a.length === 2) {
+      flags[a.slice(1)] = args[++i];
+    } else {
+      pos.push(a);
+    }
+  }
+  return { pos, flags };
+}
+
+// ── sphere generate ───────────────────────────────────────────────────────────
+
+async function cmdGenerate(pos, flags) {
+  await checkLicense();
+
+  const input  = pos[0];
+  const output = flags.output;
+
+  if (!input)  { process.stderr.write('Error: missing input CSV\n'); process.exit(1); }
+  if (!output) { process.stderr.write('Error: missing -o / --output path\n'); process.exit(1); }
+  if (!fs.existsSync(input)) {
+    process.stderr.write(`Error: file not found: ${input}\n`);
+    process.exit(1);
+  }
+
+  if (!flags.json) {
+    process.stderr.write(`Generating synthetic data from ${path.basename(input)} …\n`);
+  }
+
+  // Build args for sphere-node.js
+  const nodeArgs = [SPHERE_NODE_JS, input, output];
+  if (flags.k)           nodeArgs.push('--k',        String(flags.k));
+  if (flags.theta)       nodeArgs.push('--theta',     String(flags.theta));
+  if (flags.delta)       nodeArgs.push('--delta',     String(flags.delta));
+  if (flags['mix-prob']) nodeArgs.push('--mix-prob',  String(flags['mix-prob']));
+  if (flags.seed)        nodeArgs.push('--seed',      String(flags.seed));
+
+  return new Promise((resolve) => {
+    const proc = spawn(process.execPath, nodeArgs, { stdio: ['ignore', 'pipe', 'pipe'] });
+
+    let resultLine = '';
+    let errorLine  = '';
+
+    proc.stdout.on('data', (d) => { resultLine += d.toString(); });
+
+    proc.stderr.on('data', (d) => {
+      for (const line of d.toString().split('\n')) {
+        const s = line.trim();
+        if (!s) continue;
+        try {
+          const msg = JSON.parse(s);
+          if (typeof msg.progress === 'number' && !flags.json) {
+            progressBar(msg.progress, '');
+          } else if (msg.error) {
+            errorLine = msg.error;
+          }
+        } catch { /* ignore non-JSON lines */ }
+      }
+    });
+
+    proc.on('close', (code) => {
+      if (!flags.json) clearLine();
+
+      if (code !== 0) {
+        process.stderr.write(`Error: ${errorLine || 'sphere-node exited with code ' + code}\n`);
+        process.exit(1);
+      }
+
+      let result;
+      try { result = JSON.parse(resultLine.trim()); }
+      catch {
+        process.stderr.write('Error: could not parse sphere-node output\n');
+        process.exit(1);
+      }
+
+      if (flags.json) {
+        process.stdout.write(JSON.stringify(result) + '\n');
+        resolve();
+        return;
+      }
+
+      const elapsedS = ((result.elapsedMs || 0) / 1000).toFixed(1);
+      process.stdout.write(
+        `✓ ${output}  ${(result.rows || 0).toLocaleString()} rows \xd7 ${result.cols || '?'} cols` +
+        `  (${elapsedS} s)  seed ${result.seed}\n`,
+      );
+      if (result.idColDetected) {
+        process.stdout.write(`  ID columns excluded: ${result.idColName}\n`);
+      }
+      resolve();
+    });
+
+    proc.on('error', (err) => {
+      process.stderr.write(`Error: failed to start sphere-node: ${err.message}\n`);
+      process.exit(1);
+    });
+  });
+}
+
+// ── sphere demo ───────────────────────────────────────────────────────────────
+
+async function cmdDemo() {
+  const exampleCsv = path.join(PKG_DIR, 'examples', 'nhanes_sample.csv');
+  if (!fs.existsSync(exampleCsv)) {
+    process.stderr.write('Error: built-in example not found at ' + exampleCsv + '\n');
+    process.exit(1);
+  }
+
+  const { randomUUID } = require('crypto');
+  const tmpOut = path.join(os.tmpdir(), `sphere-demo-${randomUUID()}.csv`);
+
+  process.stdout.write('SPHERE demo — built-in NHANES dataset (4,899 rows \xd7 18 cols)\n');
+  process.stdout.write('─'.repeat(52) + '\n\n');
+
+  try {
+    await cmdGenerate([exampleCsv], { output: tmpOut });
+  } finally {
+    try { fs.unlinkSync(tmpOut); } catch {}
+    try { fs.unlinkSync(tmpOut + '.sphere.json'); } catch {}
+  }
+
+  process.stdout.write('\nTry it on your own data:\n');
+  process.stdout.write('  sphere generate your_data.csv -o synthetic.csv\n\n');
+}
+
+// ── sphere license ────────────────────────────────────────────────────────────
+
+async function cmdLicense(sub, pos, flags) {
+  if (sub === 'activate') {
+    let key = (pos[0] || flags.key || '').trim();
+    if (!key) {
+      // Simple stdin prompt (no readline dependency)
+      process.stdout.write('SPHERE license key (sphere_…): ');
+      key = await new Promise((resolve) => {
+        let buf = '';
+        process.stdin.resume();
+        process.stdin.setEncoding('utf8');
+        process.stdin.on('data', (ch) => {
+          const c = ch.toString();
+          if (c.includes('\n') || c.includes('\r')) {
+            process.stdin.pause();
+            process.stdout.write('\n');
+            resolve(buf.trim());
+          } else {
+            buf += c;
+          }
+        });
+      });
+    }
+    if (!key.startsWith('sphere_')) {
+      process.stderr.write("Error: key must start with 'sphere_'\n");
+      process.exit(1);
+    }
+    process.stdout.write('Validating …');
+    let result;
+    try { result = await validateKeyOnline(key); }
+    catch (e) {
+      process.stderr.write(`\nError: could not reach license server — ${e.message}\n`);
+      process.exit(1);
+    }
+    if (!result.valid) {
+      process.stderr.write(`\r✗ Invalid key: ${result.error || 'unknown error'}\n`);
+      process.exit(1);
+    }
+    writeLicenseKey(key);
+    writeLicenseCache(result);
+    process.stdout.write(`\r✓ License activated  —  ${result.customer || ''}\n`);
+    if (result.expiry) process.stdout.write(`  Expires: ${result.expiry}\n`);
+    return;
+  }
+
+  if (sub === 'status') {
+    const key = readStoredLicenseKey();
+    if (!key) {
+      process.stdout.write('✗ No license key configured.\n  Run:  sphere license activate <key>\n');
+      return;
+    }
+    process.stdout.write('Checking …');
+    let offline = false;
+    let result;
+    try {
+      result = await validateKeyOnline(key);
+      writeLicenseCache(result);
+    } catch {
+      result = readLicenseCache();
+      offline = true;
+      if (!result) {
+        process.stdout.write('\r✗ License server unreachable and cache expired.\n');
+        return;
+      }
+    }
+    if (result.valid) {
+      const suffix = offline ? '  (offline — cached)' : '';
+      process.stdout.write(`\r✓ License valid  —  ${result.customer || ''}${suffix}\n`);
+      if (result.expiry) process.stdout.write(`  Expires: ${result.expiry}\n`);
+    } else {
+      process.stdout.write(`\r✗ License invalid: ${result.error || 'unknown'}\n`);
+    }
+    return;
+  }
+
+  if (sub === 'clear') {
+    let removed = false;
+    for (const f of [licenseKeyFile(), licenseCacheFile()]) {
+      if (fs.existsSync(f)) { fs.unlinkSync(f); removed = true; }
+    }
+    process.stdout.write(removed ? '✓ License cleared.\n' : 'No license stored.\n');
+    return;
+  }
+
+  process.stderr.write('Usage: sphere license activate|status|clear\n');
   process.exit(1);
 }
 
-const result = spawnSync(binaryPath, process.argv.slice(2), {
-  stdio: 'inherit',
-  env: process.env,
-});
+// ── Main dispatch ─────────────────────────────────────────────────────────────
+
+async function main() {
+  const { pos, flags } = parseArgs(process.argv);
+  const cmd = pos[0];
+
+  if (flags.version || flags.v) {
+    process.stdout.write(`${VERSION}\n`);
+    process.exit(0);
+  }
+
+  if (!cmd || flags.help || flags.h) {
+    process.stdout.write(
+      `SPHERE CLI v${VERSION} — synthetic data generation\n\n` +
+      'Usage:\n' +
+      '  sphere generate <input.csv> -o <output.csv> [options]\n' +
+      '  sphere license  activate|status|clear\n' +
+      '  sphere demo\n\n' +
+      'Generate options:\n' +
+      '  --k <int>          Synthesis depth (default: 2)\n' +
+      '  --theta <float>    Rotation angle in radians (default: π/6 ≈ 0.524)\n' +
+      '  --delta <float>    Per-pair angle jitter (default: 5° ≈ 0.087)\n' +
+      '  --mix-prob <float> Privacy/utility trade-off (default: 0.75)\n' +
+      '  --seed <int>       Integer seed for reproducible output\n' +
+      '  --json             Machine-readable JSON output\n',
+    );
+    process.exit(0);
+  }
+
+  if (cmd === 'generate') {
+    await cmdGenerate(pos.slice(1), flags);
+    return;
+  }
 
-process.exit(result.status ?? 1);
+  if (cmd === 'demo') {
+    await cmdDemo();
+    return;
+  }
+
+  if (cmd === 'license') {
+    await cmdLicense(pos[1], pos.slice(2), flags);
+    return;
+  }
+
+  if (cmd === 'evaluate' || cmd === 'certify') {
+    process.stderr.write(
+      `'sphere ${cmd}' requires the Python evaluation engine, which is not included\n` +
+      'in this version. Coming soon.\n',
+    );
+    process.exit(1);
+  }
+
+  process.stderr.write(`Unknown command: ${cmd}\nRun 'sphere --help' for usage.\n`);
+  process.exit(1);
+}
+
+main().catch((err) => {
+  process.stderr.write(`\nError: ${err.message || err}\n`);
+  process.exit(1);
+});