vaultkit 0.1.0 → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md CHANGED
@@ -1,611 +1,539 @@
1
- # VaultKit
1
+ # VaultKit CLI (`vkit`)
2
2
 
3
+ [![Gem Version](https://img.shields.io/badge/gem-v0.1.0-blue.svg)](https://rubygems.org/gems/vaultkitcli)
3
4
  [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
4
- [![Build Status](https://img.shields.io/badge/build-passing-brightgreen.svg)]()
5
- [![Version](https://img.shields.io/badge/version-0.1.0-orange.svg)]()
6
- [![Go Version](https://img.shields.io/badge/go-1.21+-00ADD8.svg)](https://golang.org)
7
5
  [![Ruby Version](https://img.shields.io/badge/ruby-3.1+-CC342D.svg)](https://www.ruby-lang.org)
8
6
 
9
- > A secure, policy-driven control plane for unified data access across heterogeneous data sources.
7
+ > Command-line interface for VaultKit — secure, policy-driven data access control plane
10
8
 
11
- **TL;DR**: VaultKit prevents credential sprawl and enforces data access policies across databases. Write queries in AQL (vendor-neutral JSON), policies decide who sees what, FUNL executes safely. Schema changes require explicit reviewno silent data exposure.
9
+ The VaultKit CLI (`vkit`) is the primary interface for interacting with the VaultKit control plane. It enables users to request governed data access, track approvals, manage datasources, and deploy policy bundlesall while maintaining complete audit trails.
12
10
 
13
11
  ---
14
12
 
15
- ## 🎯 What is VaultKit?
13
+ ## 📋 Table of Contents
16
14
 
17
- VaultKit provides enterprise-grade governance and security for data access, enabling applications, engineers, and AI agents to query multiple data sources through a unified, policy-controlled interface.
18
-
19
- VaultKit is a **control plane** that governs how data is accessed across your organization. It centralizes authentication, authorization, policy evaluation, and access control—ensuring that every data request is properly authenticated, authorized, and audited before execution.
20
-
21
- ### Core Responsibilities
22
-
23
- - **Authentication & Authorization**: Identity verification and access control with RBAC/ABAC support
24
- - **Policy Evaluation**: Fine-grained policies based on user attributes, data sensitivity, geographic regions, and time constraints
25
- - **Data Masking**: Column-level masking rules (full, partial, hash) applied based on user clearance
26
- - **Connection Management**: Centralized datasource configuration and credential management
27
- - **Approval Workflows**: Require explicit approval for accessing sensitive datasets
28
- - **Audit Logging**: Complete audit trail of every data access request
29
- - **Metadata Catalog**: Automated dataset classification and schema discovery
30
- - **Session Tokens**: Short-lived, cryptographically-signed tokens for Zero-Trust security
31
- - **Schema Governance**: Git-backed policy bundles with drift detection for safe schema evolution
15
+ - [What is VaultKit CLI?](#what-is-vaultkit-cli)
16
+ - [Installation](#installation)
17
+ - [Quick Start](#quick-start)
18
+ - [Authentication](#authentication)
19
+ - [Core Workflows](#core-workflows)
20
+ - [Command Reference](#command-reference)
21
+ - [Configuration](#configuration)
22
+ - [Advanced Usage](#advanced-usage)
23
+ - [Troubleshooting](#troubleshooting)
24
+ - [Contributing](#contributing)
32
25
 
33
26
  ---
34
27
 
35
- ## What is FUNL?
28
+ ## 🎯 What is VaultKit CLI?
36
29
 
37
- FUNL (Functional Universal Query Language) is the **data plane and execution engine** that powers VaultKit's query capabilities. While VaultKit decides *if* and *how* data can be accessed, FUNL handles the actual execution.
30
+ The VaultKit CLI provides command-line access to:
38
31
 
39
- ### Core Responsibilities
32
+ | Capability | Description |
33
+ |------------|-------------|
34
+ | **Authentication** | Login, session management, identity verification |
35
+ | **Data Requests** | Submit AQL queries with automatic policy evaluation |
36
+ | **Approval Workflows** | Approve or deny access requests requiring authorization |
37
+ | **Grant Management** | Fetch data using approved, time-limited grants |
38
+ | **Datasource Admin** | Register and manage database connections (admin only) |
39
+ | **Schema Discovery** | Scan datasources and detect schema drift |
40
+ | **Policy Deployment** | Build, validate, and deploy policy bundles |
41
+ | **Audit Queries** | Search and export audit logs |
40
42
 
41
- - **AQL Translation**: Converts VaultKit's Abstract Query Language (AQL) into native query languages
42
- - **Multi-Engine Support**: Executes queries against PostgreSQL, MySQL, Snowflake, BigQuery, and more
43
- - **SQL Masking Engine**: Applies column-level masking at the SQL execution layer
44
- - **Query Execution**: Handles parameterized queries with injection prevention
45
- - **Result Sanitization**: Returns masked and filtered results back to VaultKit
46
- - **JWT Authentication**: Only accepts cryptographically-signed requests from VaultKit
43
+ ### Mental Model
47
44
 
48
- FUNL is designed to be lightweight, stateless, and horizontally scalable—making it ideal for high-throughput data access scenarios.
45
+ VaultKit uses a **request policy grant fetch** workflow:
49
46
 
50
- ---
47
+ ```
48
+ ┌──────────┐ ┌──────────┐ ┌───────┐ ┌───────┐ ┌───────────┐
49
+ │ Request │ -> │ Policy │ -> │ Grant │ -> │ Fetch │ -> │ Audit Log │
50
+ │ (AQL) │ │ Eval │ │ (JWT) │ │ Data │ │ (Record) │
51
+ └──────────┘ └──────────┘ └───────┘ └───────┘ └───────────┘
52
+ ```
51
53
 
52
- ## 🏗️ Architecture
53
-
54
- ```mermaid
55
- flowchart TB
56
- subgraph clients["🖥️ Client Layer"]
57
- app["Applications"]
58
- cli["CLI Tools"]
59
- agent["AI Agents"]
60
- end
61
-
62
- subgraph vaultkit["🛡️ VaultKit — Control Plane"]
63
- direction TB
64
- orch["Request Orchestrator"]
65
- policy["Policy Engine<br/>(ABAC/RBAC)"]
66
- auth["Authentication<br/>(JWT/SSO)"]
67
- registry["Schema Registry"]
68
- bundle["Active Policy Bundle"]
69
- conn["Connection Manager"]
70
- approval["Approval Workflow"]
71
- audit["Audit Logger"]
72
- end
73
-
74
- subgraph funl["⚡ FUNL — Data Plane"]
75
- direction TB
76
- translator["AQL Translator"]
77
- executor["Query Executor"]
78
- masking["SQL Masking Layer"]
79
- end
80
-
81
- subgraph datasources["💾 Data Sources"]
82
- pg[(PostgreSQL)]
83
- mysql[(MySQL)]
84
- snow[(Snowflake)]
85
- bq[(BigQuery)]
86
- end
87
-
88
- subgraph governance["📋 Governance Layer"]
89
- git["Git Repository<br/>(Policies & Registry)"]
90
- scan["Schema Scans"]
91
- cicd["CI/CD Pipeline"]
92
- end
93
-
94
- clients -->|AQL Request| orch
95
- orch --> auth
96
- orch --> policy
97
- policy --> bundle
98
- orch --> registry
99
- orch --> approval
100
- orch --> audit
101
- orch -->|Signed Token + AQL| translator
102
-
103
- translator --> executor
104
- executor --> masking
105
- masking --> datasources
106
- datasources --> masking
107
- masking --> executor
108
- executor --> translator
109
- translator -->|Masked Results| orch
110
- orch -->|Response| clients
111
-
112
- scan -.->|Discovers Schema| datasources
113
- scan -->|Drift Detection| registry
114
- git -->|Deploy Bundle| bundle
115
- cicd -->|Build & Validate| git
116
-
117
- style vaultkit fill:#e3f2fd
118
- style funl fill:#fff3e0
119
- style datasources fill:#f3e5f5
120
- style governance fill:#f1f8e9
121
- ```
122
-
123
- ### Component Breakdown
124
-
125
- #### VaultKit (Control Plane)
126
-
127
- - **Request Orchestrator**: Routes and manages the lifecycle of data access requests
128
- - **Policy Engine**: Evaluates ABAC/RBAC rules with support for sensitivity levels, regions, and time constraints
129
- - **Policy Priority System**: `deny` > `require_approval` > `mask` > `allow`
130
- - **Match Rules**: Dataset-based, field sensitivity, categories, specific field names
131
- - **Context Awareness**: User role, clearance level, region, environment, time windows
132
- - **Authentication Service**: Handles user identity via JWT, OAuth, or SSO integration
133
- - **Schema Registry**: Runtime database storing discovered schemas and serving as drift baseline
134
- - **Active Policy Bundle**: Immutable, Git-backed artifact containing all enforcement rules
135
- - **Connection Manager**: Manages datasource credentials (supports VaultKit store, HashiCorp Vault, AWS Secrets Manager)
136
- - **Approval Workflow**: Implements multi-stage approval for sensitive data access
137
- - **Audit Logger**: Records every request to pluggable sinks (SQLite, PostgreSQL, S3, etc.)
138
-
139
- #### FUNL (Data Plane)
140
- - **AQL Translator**: Parses AQL and generates engine-specific SQL with proper escaping
141
- - **Query Executor**: Maintains connection pools and executes parameterized queries
142
- - **SQL Masking Layer**: Injects masking functions into SELECT statements based on policy
143
- - **Engine Plugins**: Extensible architecture for adding new database engines
144
-
145
- #### Governance Layer
146
- - **Git Repository**: Source of truth for all policies and dataset registries
147
- - **Schema Scans**: Observational tooling that discovers database changes
148
- - **CI/CD Pipeline**: Automated validation, bundling, and deployment of policy changes
54
+ **Key Principles:**
55
+ - Users **request** data, not queries
56
+ - Policies **decide** what happens (allow, mask, deny, require approval)
57
+ - Grants **authorize** execution with TTLs
58
+ - FUNL **executes** queries safely
59
+ - Everything is **audited**
149
60
 
150
61
  ---
151
62
 
152
- ## 📋 Schema Discovery & Policy Management
63
+ ## 📦 Installation
153
64
 
154
- VaultKit's approach to schema evolution is designed for **security, auditability, and safe change management**.
65
+ ### Prerequisites
155
66
 
156
- ### The Core Philosophy: Intent vs. Reality
67
+ - **Ruby** 3.1 or higher
68
+ - **VaultKit Control Plane** running and accessible
157
69
 
158
- VaultKit deliberately separates:
70
+ ### Option 1: Install from RubyGems (Recommended)
159
71
 
160
- - **Policy Bundles** (Intent) — Git-backed, immutable definitions of *what should be allowed*
161
- - **Schema Scans** (Reality) — Observational discovery of *what actually exists*
72
+ ```bash
73
+ gem install vaultkitcli
74
+ ```
162
75
 
163
- This separation prevents silent data exposure. When new tables or sensitive columns appear, they require **explicit policy review** before access is granted.
76
+ ### Option 2: Install from Source
164
77
 
165
- ```mermaid
166
- flowchart LR
167
- A[Live Database] -->|1. Scan Discovers| B[Schema Changes]
168
- B -->|2. Drift Detection| C[Schema Registry]
169
- C -->|3. Human Review| D[Policy Updates in Git]
170
- D -->|4. CI Validation| E[Build Bundle]
171
- E -->|5. Deploy| F[Active Bundle]
172
- F -->|Enforces| G[Access Control]
173
-
174
- style A fill:#f3e5f5
175
- style C fill:#fff3e0
176
- style D fill:#e8f5e9
177
- style F fill:#e3f2fd
178
- style G fill:#fce4ec
78
+ ```bash
79
+ git clone https://github.com/yourorg/vaultkit-cli.git
80
+ cd vaultkit-cli
81
+ bundle install
82
+ gem build vaultkitcli.gemspec
83
+ gem install ./vaultkitcli-0.1.0.gem
179
84
  ```
180
85
 
181
- ### How It Works
86
+ ### Verify Installation
182
87
 
183
- **Policy Bundles** are versioned artifacts that contain:
184
- - Dataset registry (expected schema)
185
- - Datasource definitions
186
- - Compiled policy rules
187
- - Bundle metadata (version, checksum, source)
88
+ ```bash
89
+ vkit --version
90
+ # => VaultKit CLI v0.1.0
188
91
 
189
- **Schema Scans** perform these steps:
190
- 1. Introspect datasources via FUNL
191
- 2. Discover tables and columns
192
- 3. Classify fields (PII, financial, internal)
193
- 4. Compute diff against runtime registry
194
- 5. Surface changes for review
92
+ vkit --help
93
+ ```
195
94
 
196
- **Key Safety Principle:**
95
+ ---
197
96
 
198
- > Scans inform. Bundles enforce. Humans decide.
97
+ ## Quick Start
199
98
 
200
- Scans **never** automatically update active policies. This ensures:
201
- - No silent expansion of policy scope
202
- - No auto-granting access to new sensitive data
203
- - Full Git-based audit trail
204
- - Compliance with SOC2, ISO 27001, GDPR, HIPAA
99
+ ### 1. Set Environment Variables
205
100
 
206
- ### Quick Example: Handling Schema Drift
101
+ ```bash
102
+ export VKIT_API_URL=http://localhost:3000
103
+ ```
207
104
 
105
+ For production:
208
106
  ```bash
209
- # 1. Discover schema changes (safe, read-only)
210
- vkit scan production_db
107
+ export VKIT_API_URL=https://vaultkit.company.com
108
+ ```
211
109
 
212
- # Output shows drift:
213
- # + dataset: customers
214
- # + field: ssn (string) [PII] ⚠️ NEW - requires policy
215
- # ~ field: email (text → varchar) [PII] ✓ Already masked
110
+ ### 2. Authenticate
216
111
 
217
- # 2. Review and update baseline
218
- vkit scan production_db --apply
112
+ ```bash
113
+ vkit login --endpoint http://localhost:3000 --email analyst@company.com
114
+ ```
219
115
 
220
- # 3. Update policies in Git
221
- vim policies/customer_data.yaml
116
+ You'll receive a prompt for your password or be redirected to SSO.
222
117
 
223
- # 4. Build and deploy new bundle
224
- vkit policy bundle
225
- vkit policy deploy
118
+ ### 3. Submit Your First Request
119
+
120
+ ```bash
121
+ vkit request --aql '{
122
+ "source_table": "customers",
123
+ "columns": ["id", "email", "total_spend"],
124
+ "limit": 10
125
+ }'
226
126
  ```
227
127
 
228
- ### CI/CD Integration (Recommended)
128
+ ### 4. Fetch Results
229
129
 
230
- ```yaml
231
- name: Schema Drift Check
232
- on: [schedule, push]
233
- jobs:
234
- drift-detection:
235
- runs-on: ubuntu-latest
236
- steps:
237
- - name: Scan all datasources
238
- run: vkit scan --all --fail-on-drift
239
-
240
- - name: Create PR if drift detected
241
- if: failure()
242
- run: |
243
- vkit scan --all --diff > drift-report.md
244
- gh pr create --title "Schema Drift Detected" --body-file drift-report.md
130
+ If granted immediately:
131
+ ```bash
132
+ vkit fetch --grant g_customers_abc123xyz
245
133
  ```
246
134
 
247
- 📘 **[Full Bundling & Scan System Documentation →](./docs/BUNDLING_SCAN_SYSTEM.md)**
135
+ If approval required:
136
+ ```bash
137
+ # Check status
138
+ vkit requests list --state pending
139
+
140
+ # After approval
141
+ vkit fetch --grant g_customers_abc123xyz
142
+ ```
248
143
 
249
144
  ---
250
145
 
251
- ## 🔄 AQL to Native Query Translation
146
+ ## 🔐 Authentication
147
+
148
+ ### `vkit login`
149
+
150
+ Authenticate with the VaultKit control plane.
151
+
152
+ ```bash
153
+ vkit login --endpoint http://localhost:3000 --email analyst@acme.com
154
+ ```
252
155
 
253
- AQL (Access Query Language) is a structured JSON format that describes **what** to fetch, not **how**. This abstraction allows VaultKit to enforce policies consistently across different database engines.
156
+ **Options:**
157
+ - `--endpoint` — VaultKit API endpoint (can also use `VKIT_API_URL`)
158
+ - `--email` — Email address for authentication
254
159
 
255
- ### AQL Structure
160
+ **SSO Integration:**
256
161
 
257
- AQL is a structured JSON specification with the following top-level fields:
162
+ If your organization uses SSO, you'll be redirected to your identity provider:
258
163
 
259
- ```json
260
- {
261
- "source_table": "table_name",
262
- "columns": ["field1", "field2"],
263
- "joins": [],
264
- "aggregates": [],
265
- "filters": [],
266
- "group_by": [],
267
- "having": [],
268
- "order_by": null,
269
- "limit": 0,
270
- "offset": 0
271
- }
164
+ ```bash
165
+ vkit login --endpoint https://vaultkit.company.com --email you@company.com
166
+
167
+ # Output:
168
+ # Opening browser for SSO authentication...
169
+ # ✓ Authentication successful
170
+ # Session expires: 2024-01-15 18:00:00 UTC
272
171
  ```
273
172
 
274
- ### Translation Examples
173
+ ### `vkit whoami`
275
174
 
276
- **Simple Query:**
277
- ```json
278
- {
279
- "source_table": "customers",
280
- "columns": ["email", "country", "revenue"],
281
- "filters": [
282
- { "field": "country", "operator": "eq", "value": "US" },
283
- { "field": "revenue", "operator": "gt", "value": 10000 }
284
- ],
285
- "limit": 100
286
- }
287
- ```
288
-
289
- **FUNL Translation → PostgreSQL:**
290
- ```sql
291
- SELECT email, country, revenue
292
- FROM customers
293
- WHERE country = $1 AND revenue > $2
294
- LIMIT 100
295
- ```
296
-
297
- **FUNL Translation → MySQL:**
298
- ```sql
299
- SELECT `email`, `country`, `revenue`
300
- FROM `customers`
301
- WHERE `country` = ? AND `revenue` > ?
302
- LIMIT 100
303
- ```
304
-
305
- **Complex Query with JOINs and Aggregates:**
306
- ```json
307
- {
308
- "source_table": "users",
309
- "joins": [
310
- {
311
- "type": "LEFT",
312
- "table": "orders",
313
- "left_field": "users.id",
314
- "right_field": "orders.user_id"
315
- }
316
- ],
317
- "columns": ["users.email", "users.username"],
318
- "aggregates": [
319
- {
320
- "func": "sum",
321
- "field": "orders.amount",
322
- "alias": "total_spent"
323
- }
324
- ],
325
- "group_by": ["users.email", "users.username"],
326
- "having": [
327
- {
328
- "operator": "gt",
329
- "field": "SUM(orders.amount)",
330
- "value": 500
331
- }
332
- ],
333
- "order_by": {
334
- "column": "total_spent",
335
- "direction": "DESC"
336
- },
337
- "limit": 10
338
- }
175
+ Display the currently authenticated user and session information.
176
+
177
+ ```bash
178
+ vkit whoami
339
179
  ```
340
180
 
341
- **FUNL Translation → PostgreSQL:**
342
- ```sql
343
- SELECT users.email, users.username, SUM(orders.amount) AS total_spent
344
- FROM users
345
- LEFT JOIN orders ON users.id = orders.user_id
346
- GROUP BY users.email, users.username
347
- HAVING SUM(orders.amount) > $1
348
- ORDER BY total_spent DESC
349
- LIMIT 10
181
+ **Output:**
182
+ ```
183
+ User ID: user_12345
184
+ Email: analyst@company.com
185
+ Role: analyst
186
+ Clearance Level: 2 (high)
187
+ Region: US
188
+ Session Expires: 2024-01-15 18:00:00 UTC
350
189
  ```
351
190
 
352
- ### Advanced Features
191
+ ### `vkit logout`
192
+
193
+ Clear stored credentials and end session.
353
194
 
354
- **Complex Filtering with OR Logic:**
355
- ```json
356
- {
357
- "source_table": "users",
358
- "columns": ["id", "email", "role"],
359
- "filters": [
360
- {
361
- "logic": "OR",
362
- "conditions": [
363
- { "field": "users.role", "operator": "eq", "value": "admin" },
364
- { "field": "users.role", "operator": "eq", "value": "manager" }
365
- ]
366
- },
367
- { "field": "orders.amount", "operator": "gt", "value": 100 }
368
- ]
369
- }
195
+ ```bash
196
+ vkit logout
370
197
  ```
371
198
 
372
- **Translation:**
373
- ```sql
374
- SELECT id, email, role
375
- FROM users
376
- WHERE (users.role = $1 OR users.role = $2) AND orders.amount > $3
199
+ ---
200
+
201
+ ## 🔄 Core Workflows
202
+
203
+ ### Workflow 1: Immediate Access (Auto-Approved)
204
+
205
+ ```bash
206
+ # 1. Submit request
207
+ vkit request --aql '{
208
+ "source_table": "public_reports",
209
+ "columns": ["report_id", "title", "published_date"],
210
+ "limit": 10
211
+ }'
212
+
213
+ # Output:
214
+ # ✓ Request granted immediately
215
+ # Grant ID: g_customers_abc123xyz
216
+ # TTL: 4 hours
217
+ # Use: vkit fetch --grant g_customers_abc123xyz
218
+
219
+ # 2. Fetch data
220
+ vkit fetch --grant g_customers_abc123xyz --format table
377
221
  ```
378
222
 
379
- **Supported Operators:**
380
- - Comparison: `eq`, `neq`, `gt`, `lt`, `gte`, `lte`
381
- - Pattern matching: `like`
382
- - Set operations: `in`
383
- - Null checks: `is_null`, `is_not_null`
223
+ ### Workflow 2: Approval Required
384
224
 
385
- **Supported Aggregations:**
386
- - `sum`, `count`, `avg`, `min`, `max`
225
+ ```bash
226
+ # 1. Submit request
227
+ vkit request --aql '{
228
+ "source_table": "financial_transactions",
229
+ "columns": ["transaction_id", "amount", "customer_id"],
230
+ "filters": [{"field": "amount", "operator": "gt", "value": 10000}]
231
+ }'
387
232
 
388
- **Supported JOINs:**
389
- - `INNER`, `LEFT`, `RIGHT`, `FULL`
233
+ # Output:
234
+ # Approval required
235
+ # Request ID: req_20240115_xyz789
236
+ # Approver: finance_manager
237
+ # Reason: Financial data requires manager approval
238
+ # Status: PENDING
390
239
 
391
- ### SQL-Level Masking
240
+ # 2. Check status
241
+ vkit requests list --state pending
392
242
 
393
- FUNL's **Masking Dialect System** applies transformations directly in SQL based on VaultKit's policy decisions:
243
+ # 3. After approval, fetch data
244
+ vkit fetch --grant g_customers_approved_abc123
245
+ ```
394
246
 
395
- | Masking Type | PostgreSQL | MySQL | Snowflake |
396
- |-------------|------------|-------|-----------|
397
- | **Full** | `'*****' AS email` | `'*****' AS email` | `'*****' AS email` |
398
- | **Partial** | `CONCAT(LEFT(email, 3), '****')` | `CONCAT(LEFT(email, 3), '****')` | `CONCAT(LEFT(email, 3), '****')` |
399
- | **Hash** | `ENCODE(SHA256(email::bytea), 'hex')` | `SHA2(email, 256)` | `SHA2(email, 256)` |
247
+ ### Workflow 3: Denied Access
400
248
 
401
- ---
249
+ ```bash
250
+ vkit request --aql '{
251
+ "source_table": "payment_cards",
252
+ "columns": ["card_number", "cvv"],
253
+ "limit": 10
254
+ }'
402
255
 
403
- ## ✨ Key Features
404
-
405
- ### VaultKit (Control Plane)
406
-
407
- | Feature | Description |
408
- |---------|-------------|
409
- | **AQL Orchestration** | Vendor-neutral query language prevents SQL injection and enables policy enforcement |
410
- | **Policy Engine** | Attribute-based access control with support for clearance levels, regions, sensitivity tags, and time windows |
411
- | **Policy Priority System** | Hierarchical decision making: `deny` > `require_approval` > `mask` > `allow` |
412
- | **Field-Level Policies** | Match rules based on dataset name, field sensitivity, categories (pii, financial, etc.), or specific field names |
413
- | **Context-Aware Rules** | Policies evaluate requester role, clearance level, region, environment, and time constraints |
414
- | **Approval Workflows** | Require manager/security approval for accessing PII, financial data, or production environments |
415
- | **Zero-Trust Sessions** | Short-lived, cryptographically-signed tokens with automatic expiration |
416
- | **Credential Abstraction** | Never expose database credentials to users—supports multiple secret backends |
417
- | **Git-Backed Governance** | All policies and registries versioned in Git with full audit trail |
418
- | **Schema Drift Detection** | Automated discovery of database changes with manual approval workflow |
419
- | **Comprehensive Auditing** | Every query logged with user identity, timestamp, policy decisions, and results metadata |
420
- | **CLI & SDK** | Rich command-line tools (`vkit`) and programmatic SDK for integration |
421
-
422
- ### FUNL (Data Plane)
423
-
424
- | Feature | Description |
425
- |---------|-------------|
426
- | **Multi-Engine Translation** | Supports PostgreSQL, MySQL, Snowflake, BigQuery with identical AQL interface |
427
- | **SQL-Level Masking** | Masking applied during query execution—no post-processing overhead |
428
- | **Injection Prevention** | All queries use parameterized execution with proper type binding |
429
- | **Raw SQL Support** | Supports raw SQL for schema introspection and admin operations (policy-controlled) |
430
- | **Horizontal Scaling** | Stateless design enables running multiple FUNL instances behind a load balancer |
431
- | **JWT Verification** | Only executes queries signed by VaultKit's private key |
256
+ # Output:
257
+ # ❌ Request denied
258
+ # Reason: Direct payment card data access forbidden
259
+ # Policy: pci_dss_restrictions
260
+ # Alternative: Use tokenized_card_id field instead
261
+ # Audit ID: audit_20240115_denied_123
262
+ ```
432
263
 
433
264
  ---
434
265
 
435
- ## 🚀 Getting Started
436
-
437
- ### Prerequisites Checklist
266
+ ## 📚 Command Reference
438
267
 
439
- Before installing VaultKit, ensure you have:
268
+ ### Authentication Commands
440
269
 
441
- - [ ] **Ruby** 3.1+ installed (`ruby --version`)
442
- - [ ] **Go** 1.21+ installed (`go version`)
443
- - [ ] **OpenSSL** for key generation
444
- - [ ] **Git** for policy management
445
- - [ ] Access credentials for at least one supported datasource
446
- - [ ] (Optional) **Docker** for containerized FUNL deployment
270
+ #### `vkit login`
447
271
 
448
- ### Quick Start (5 Minutes)
272
+ Authenticate with VaultKit.
449
273
 
450
- This streamlined path gets you querying data quickly. For production deployments, see [Detailed Setup](#detailed-setup) below.
274
+ ```bash
275
+ vkit login --endpoint <URL> --email <EMAIL>
276
+ ```
451
277
 
452
- #### 1. Generate Cryptographic Keys
278
+ **Options:**
279
+ - `--endpoint, -e` — VaultKit API endpoint (default: `$VKIT_API_URL`)
280
+ - `--email` — User email address
281
+ - `--password` — Password (prompted if not provided)
453
282
 
283
+ **Examples:**
454
284
  ```bash
455
- # Generate RSA key pair for token signing
456
- openssl genpkey -algorithm RSA -out vaultkit_private.pem -pkeyopt rsa_keygen_bits:2048
457
- openssl rsa -pubout -in vaultkit_private.pem -out vaultkit_public.pem
285
+ # Interactive login
286
+ vkit login --endpoint http://localhost:3000 --email analyst@company.com
458
287
 
459
- # Set environment variables
460
- export VKIT_PRIVATE_KEY="$(pwd)/vaultkit_private.pem"
461
- export VKIT_PUBLIC_KEY="$(pwd)/vaultkit_public.pem"
462
- export FUNL_PUBLIC_KEY="$(pwd)/vaultkit_public.pem"
288
+ # Non-interactive (CI/CD)
289
+ vkit login --endpoint https://vaultkit.company.com --email bot@company.com --password $BOT_PASSWORD
463
290
  ```
464
291
 
465
- ⚠️ **Security Note**: Keep your private key secure. Add `*.pem` to your `.gitignore`.
292
+ #### `vkit whoami`
466
293
 
467
- #### 2. Start FUNL
294
+ Display current user information.
468
295
 
469
296
  ```bash
470
- # Using Docker (recommended)
471
- export FUNL_URL="http://localhost:8080"
472
- docker compose up -d
297
+ vkit whoami [--format json|table]
298
+ ```
473
299
 
474
- # Verify FUNL is running
475
- curl http://localhost:8080/health
300
+ **Examples:**
301
+ ```bash
302
+ vkit whoami
303
+ vkit whoami --format json
476
304
  ```
477
305
 
478
- #### 3. Install VaultKit CLI
306
+ #### `vkit logout`
307
+
308
+ End session and clear credentials.
479
309
 
480
310
  ```bash
481
- git clone https://github.com/yourorg/vaultkitcli.git
482
- cd vaultkitcli
483
- gem build vaultkitcli.gemspec
484
- gem install ./vaultkitcli-0.1.0.gem
311
+ vkit logout
312
+ ```
485
313
 
486
- # Verify installation
487
- vkit --version
314
+ ---
315
+
316
+ ### Data Request Commands
317
+
318
+ #### `vkit request`
319
+
320
+ Submit a governed data request using AQL.
321
+
322
+ ```bash
323
+ vkit request --aql <AQL_JSON> [OPTIONS]
488
324
  ```
489
325
 
490
- #### 4. Initialize & Connect
326
+ **Options:**
327
+ - `--aql` — AQL JSON payload (inline or via STDIN)
328
+ - `--env` — Environment (default: `production`)
329
+ - `--requester_region` — Your geographic region (e.g., `US`, `EU`)
330
+ - `--dataset_region` — Dataset's geographic region
331
+ - `--requester_clearance` — Clearance level: `low`, `high`, `admin`
332
+ - `--format` — Output format: `json`, `table` (default: `table`)
491
333
 
334
+ **Inline AQL:**
492
335
  ```bash
493
- # Authenticate
494
- vkit login
336
+ vkit request \
337
+ --aql '{"source_table":"customers","columns":["email","total_spend"],"limit":100}' \
338
+ --requester_region EU \
339
+ --dataset_region EU \
340
+ --requester_clearance high
341
+ ```
495
342
 
496
- # Register your first datasource
497
- vkit datasource add \
498
- --id demo_db \
499
- --engine postgres \
500
- --username readonly_user \
501
- --password $DB_PASSWORD \
502
- --config '{
503
- "host": "localhost",
504
- "port": 5432,
505
- "database": "analytics"
506
- }'
343
+ **AQL via STDIN:**
344
+ ```bash
345
+ cat query.json | vkit request
346
+
347
+ # Or
348
+ echo '{"source_table":"orders","columns":["order_id","total"]}' | vkit request
349
+ ```
507
350
 
508
- # Discover schema
509
- vkit scan demo_db --apply
351
+ **AQL from File:**
352
+ ```bash
353
+ vkit request --aql "$(cat queries/customer_analysis.json)"
510
354
  ```
511
355
 
512
- #### 5. Query Data
356
+ **Possible Outcomes:**
357
+
358
+ | Outcome | Description | Next Step |
359
+ |---------|-------------|-----------|
360
+ | ✅ **Granted** | Immediate access | `vkit fetch --grant <GRANT_ID>` |
361
+ | ⏳ **Queued** | Requires approval | Wait for approval, check with `vkit requests list` |
362
+ | ❌ **Denied** | Policy disallows | Review policy restrictions |
363
+
364
+ #### `vkit requests list`
365
+
366
+ View your request history.
513
367
 
514
368
  ```bash
515
- # Submit AQL query
516
- vkit request --datasource demo_db --aql '{
517
- "source_table": "users",
518
- "columns": ["id", "email", "created_at"],
519
- "limit": 10
520
- }'
369
+ vkit requests list [--state STATE] [--format FORMAT]
370
+ ```
371
+
372
+ **Options:**
373
+ - `--state` — Filter by state: `all`, `pending`, `approved`, `denied` (default: `all`)
374
+ - `--format` — Output format: `json`, `table` (default: `table`)
375
+ - `--limit` — Number of results (default: 50)
376
+
377
+ **Examples:**
378
+ ```bash
379
+ # All requests
380
+ vkit requests list
381
+
382
+ # Only pending requests
383
+ vkit requests list --state pending
384
+
385
+ # Approved requests in JSON
386
+ vkit requests list --state approved --format json
521
387
 
522
- # Fetch results (use grant ID from previous command)
523
- vkit fetch --grant grant_abc123xyz
388
+ # Last 100 requests
389
+ vkit requests list --limit 100
390
+ ```
391
+
392
+ #### `vkit requests show`
393
+
394
+ Show details for a specific request.
395
+
396
+ ```bash
397
+ vkit requests show <REQUEST_ID>
524
398
  ```
525
399
 
526
- 🎉 **You're now querying data through VaultKit!** Continue to [Detailed Setup](#detailed-setup) for production configuration.
400
+ **Example:**
401
+ ```bash
402
+ vkit requests show req_20240115_xyz789
403
+ ```
527
404
 
528
405
  ---
529
406
 
530
- ### Detailed Setup
407
+ ### Approval Commands
531
408
 
532
- #### Installation Options
409
+ #### `vkit approval:list`
533
410
 
534
- **Option A: From Source (Recommended for Development)**
411
+ List approval requests you are authorized to act on.
535
412
 
536
413
  ```bash
537
- # Clone and install CLI
538
- git clone https://github.com/yourorg/vaultkitcli.git
539
- cd vaultkitcli
540
- bundle install
541
- gem build vaultkitcli.gemspec
542
- gem install ./vaultkitcli-0.1.0.gem
414
+ vkit approval:list [--state STATE] [--format FORMAT]
415
+ ```
543
416
 
544
- # Clone and build FUNL
545
- git clone https://github.com/yourorg/funl.git
546
- cd funl
547
- go build -o funl ./cmd/funl
548
- ./funl serve --port 8080
417
+ **Options:**
418
+ - `--state` — Filter by state: `all`, `pending`, `approved`, `denied` (default: `pending`)
419
+ - `--format` — Output format: `json`, `table` (default: `table`)
420
+
421
+ **Examples:**
422
+ ```bash
423
+ # Pending approvals
424
+ vkit approval:list
425
+
426
+ # All approvals you've handled
427
+ vkit approval:list --state all
428
+
429
+ # JSON output for scripting
430
+ vkit approval:list --format json
549
431
  ```
550
432
 
551
- **Option B: Using Docker (Recommended for Production)**
433
+ #### `vkit approval:approve`
434
+
435
+ Approve a pending request.
552
436
 
553
437
  ```bash
554
- # docker-compose.yml
555
- version: '3.8'
556
- services:
557
- funl:
558
- image: vaultkit/funl:0.1.0
559
- ports:
560
- - "8080:8080"
561
- environment:
562
- - FUNL_PUBLIC_KEY=/keys/vaultkit_public.pem
563
- volumes:
564
- - ./vaultkit_public.pem:/keys/vaultkit_public.pem:ro
565
- healthcheck:
566
- test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
567
- interval: 10s
568
- timeout: 3s
569
- retries: 3
438
+ vkit approval:approve <REQUEST_ID> [--ttl SECONDS] [--notes "..."]
570
439
  ```
571
440
 
572
- #### Environment Configuration
441
+ **Options:**
442
+ - `--ttl` — Grant lifetime in seconds (default: 3600)
443
+ - `--notes` — Approval notes (optional)
573
444
 
574
- Create a configuration file for persistent settings:
445
+ **Examples:**
446
+ ```bash
447
+ # Basic approval
448
+ vkit approval:approve 42
449
+
450
+ # With custom TTL and notes
451
+ vkit approval:approve 42 \
452
+ --ttl 7200 \
453
+ --notes "Approved for Q4 analysis. Limited to aggregate data only."
454
+ ```
455
+
456
+ #### `vkit approval:deny`
457
+
458
+ Deny a pending request.
575
459
 
576
460
  ```bash
577
- # ~/.vkit/config.yaml
578
- funl_url: "http://localhost:8080"
579
- private_key_path: "/path/to/vaultkit_private.pem"
580
- public_key_path: "/path/to/vaultkit_public.pem"
581
- default_datasource: "production_db"
582
- audit_log_path: "/var/log/vaultkit/audit.log"
461
+ vkit approval:deny <REQUEST_ID> --reason "..."
583
462
  ```
584
463
 
585
- Add environment variables to your shell profile:
464
+ **Options:**
465
+ - `--reason` — Required explanation for denial
586
466
 
467
+ **Examples:**
587
468
  ```bash
588
- # ~/.bashrc or ~/.zshrc
589
- export VKIT_PRIVATE_KEY="/path/to/vaultkit_private.pem"
590
- export VKIT_PUBLIC_KEY="/path/to/vaultkit_public.pem"
591
- export FUNL_PUBLIC_KEY="/path/to/vaultkit_public.pem"
592
- export FUNL_URL="http://localhost:8080"
593
- export VKIT_CONFIG_PATH="$HOME/.vkit/config.yaml"
469
+ vkit approval:deny 42 --reason "Insufficient business justification"
470
+
471
+ vkit approval:deny 42 --reason "Request violates data minimization policy"
594
472
  ```
595
473
 
596
- #### Datasource Configuration
474
+ ---
597
475
 
598
- VaultKit supports multiple credential storage backends:
476
+ ### Data Fetch Commands
477
+
478
+ #### `vkit fetch`
479
+
480
+ Fetch data using a previously issued grant.
599
481
 
600
- **Built-in Storage (Development):**
482
+ ```bash
483
+ vkit fetch --grant <GRANT_REF> [--format FORMAT]
484
+ ```
485
+
486
+ **Options:**
487
+ - `--grant, -g` — Grant reference or ID (required)
488
+ - `--format` — Output format: `json`, `table`, `csv` (default: `table`)
489
+ - `--output, -o` — Write to file instead of STDOUT
490
+
491
+ **Examples:**
492
+ ```bash
493
+ # Fetch and display as table
494
+ vkit fetch --grant g_customers_abc123xyz
495
+
496
+ # Fetch as JSON
497
+ vkit fetch --grant g_customers_abc123xyz --format json
498
+
499
+ # Fetch and save to CSV
500
+ vkit fetch --grant g_customers_abc123xyz --format csv --output results.csv
501
+
502
+ # Pipe to jq for processing
503
+ vkit fetch --grant g_customers_abc123xyz --format json | jq '.data[] | select(.revenue > 1000)'
504
+ ```
505
+
506
+ ---
507
+
508
+ ### Datasource Management (Admin Only)
509
+
510
+ #### `vkit datasource add`
511
+
512
+ Register a new datasource.
513
+
514
+ ```bash
515
+ vkit datasource add --id <ID> --engine <ENGINE> [OPTIONS]
516
+ ```
517
+
518
+ **Options:**
519
+ - `--id` — Unique datasource identifier (required)
520
+ - `--engine` — Database engine: `postgres`, `mysql`, `snowflake`, `bigquery` (required)
521
+ - `--username` — Database username
522
+ - `--password` — Database password
523
+ - `--config` — JSON configuration (host, port, database, etc.)
524
+ - `--credential-backend` — Secret backend: `vaultkit`, `vault`, `aws-secrets`
525
+
526
+ **Examples:**
527
+
528
+ **PostgreSQL:**
601
529
  ```bash
602
530
  vkit datasource add \
603
- --id staging_db \
531
+ --id production_pg \
604
532
  --engine postgres \
605
- --username app_user \
533
+ --username vaultkit_ro \
606
534
  --password $DB_PASSWORD \
607
535
  --config '{
608
- "host": "staging.db.internal",
536
+ "host": "db.production.internal",
609
537
  "port": 5432,
610
538
  "database": "analytics",
611
539
  "ssl_mode": "require",
@@ -613,349 +541,669 @@ vkit datasource add \
613
541
  }'
614
542
  ```
615
543
 
616
- **HashiCorp Vault (Production):**
544
+ **With HashiCorp Vault:**
617
545
  ```bash
618
546
  vkit datasource add \
619
- --id production_db \
547
+ --id production_pg \
620
548
  --engine postgres \
621
549
  --credential-backend vault \
622
550
  --vault-path secret/data/databases/production \
623
551
  --config '{
624
- "host": "prod.db.internal",
552
+ "host": "db.production.internal",
625
553
  "port": 5432,
626
554
  "database": "analytics",
627
555
  "ssl_mode": "verify-full"
628
556
  }'
629
557
  ```
630
558
 
631
- **AWS Secrets Manager:**
559
+ **Snowflake:**
632
560
  ```bash
633
561
  vkit datasource add \
634
- --id cloud_db \
635
- --engine postgres \
636
- --credential-backend aws-secrets \
637
- --secret-arn arn:aws:secretsmanager:us-east-1:123456789:secret:db-creds \
562
+ --id snowflake_prod \
563
+ --engine snowflake \
564
+ --username analytics_user \
565
+ --password $SNOWFLAKE_PASSWORD \
638
566
  --config '{
639
- "host": "rds.amazonaws.com",
640
- "port": 5432,
641
- "database": "production"
567
+ "account": "xy12345.us-east-1",
568
+ "warehouse": "ANALYTICS_WH",
569
+ "database": "PRODUCTION",
570
+ "schema": "PUBLIC"
642
571
  }'
643
572
  ```
644
573
 
645
- #### Policy Setup
574
+ #### `vkit datasource list`
646
575
 
647
- Initialize your policy repository:
576
+ List all registered datasources.
648
577
 
649
578
  ```bash
650
- # Create policy repository structure
651
- mkdir -p vaultkit-policies/{policies,registries,datasources}
652
- cd vaultkit-policies
579
+ vkit datasource list [--format FORMAT]
580
+ ```
581
+
582
+ **Examples:**
583
+ ```bash
584
+ vkit datasource list
585
+ vkit datasource list --format json
586
+ ```
653
587
 
654
- git init
655
- git add .
656
- git commit -m "Initialize VaultKit policies"
588
+ #### `vkit datasource get`
657
589
 
658
- # Link VaultKit to your policy repo
659
- vkit policy init --repo $(pwd)
590
+ Get details for a specific datasource.
591
+
592
+ ```bash
593
+ vkit datasource get <DATASOURCE_ID>
660
594
  ```
661
595
 
662
- Create your first policy (`policies/base_access.yaml`):
596
+ **Example:**
597
+ ```bash
598
+ vkit datasource get production_pg
599
+ ```
663
600
 
664
- ```yaml
665
- id: analyst_basic_access
666
- description: Basic read access for analysts with PII masking
667
- priority: 100
601
+ #### `vkit datasource test`
668
602
 
669
- match:
670
- datasets:
671
- - customers
672
- - orders
673
-
674
- context:
675
- requester_role: analyst
676
- environment: production
603
+ Test connectivity to a datasource.
677
604
 
678
- rules:
679
- - field_category: pii
680
- action: mask
681
- mask_type: partial
682
- reason: "PII must be masked for analysts"
683
-
684
- - field_category: financial
685
- action: require_approval
686
- approver_role: finance_manager
687
- ttl: "2h"
688
- reason: "Financial data requires approval"
689
-
690
- - default:
691
- action: allow
692
- ttl: "8h"
605
+ ```bash
606
+ vkit datasource test <DATASOURCE_ID>
693
607
  ```
694
608
 
695
- Build and deploy the policy bundle:
609
+ **Example:**
610
+ ```bash
611
+ vkit datasource test production_pg
612
+
613
+ # Output:
614
+ # Testing connection to production_pg...
615
+ # ✓ Connection successful
616
+ # Engine: PostgreSQL 15.3
617
+ # Latency: 45ms
618
+ ```
619
+
620
+ #### `vkit datasource remove`
621
+
622
+ Remove a datasource (admin only).
696
623
 
697
624
  ```bash
698
- # Validate policies
699
- vkit policy validate
625
+ vkit datasource remove <DATASOURCE_ID> [--force]
626
+ ```
700
627
 
701
- # Build bundle
702
- vkit policy bundle --output bundle.tar.gz
628
+ **Options:**
629
+ - `--force` Skip confirmation prompt
703
630
 
704
- # Deploy to active control plane
705
- vkit policy deploy --bundle bundle.tar.gz
631
+ **Example:**
632
+ ```bash
633
+ vkit datasource remove old_staging_customers_db --force
706
634
  ```
707
635
 
708
636
  ---
709
637
 
710
- ## 🎯 Use Cases
638
+ ### Schema Discovery Commands
711
639
 
712
- ### 1. **Secure AI Agent Access**
713
- Enable LLM-powered agents to query production databases without exposing credentials. VaultKit enforces policies that restrict which tables and columns AI agents can access, with automatic masking of sensitive fields.
640
+ #### `vkit scan`
714
641
 
715
- **Problem:**
716
- - AI agents need data context for decision-making
717
- - Direct database access creates security risks
718
- - Credentials in AI systems can leak
719
- - Uncontrolled queries can expose sensitive data
642
+ Scan a datasource and detect schema drift.
720
643
 
721
- **VaultKit Solution:**
644
+ ```bash
645
+ vkit scan <DATASOURCE_ID> [--mode MODE]
646
+ ```
722
647
 
723
- ```yaml
724
- # policies/ai_agent_restrictions.yaml
725
- id: ai_agent_restrictions
726
- match:
727
- fields:
728
- category: pii
648
+ **Options:**
649
+ - `--mode` — Scan mode: `diff`, `apply` (default: `diff`)
650
+ - `--format` — Output format: `json`, `table` (default: `table`)
729
651
 
730
- context:
731
- requester_role: ai_agent
732
- environment: production
652
+ **Modes:**
733
653
 
734
- action:
735
- mask: true
736
- mask_type: hash
737
- reason: "PII must be hashed for AI agents"
738
- ttl: "15m"
654
+ | Mode | Description |
655
+ |------|-------------|
656
+ | `diff` | Show changes without applying (safe, read-only) |
657
+ | `apply` | Update baseline registry with discovered schema |
658
+
659
+ **Examples:**
660
+
661
+ **Discover Changes:**
662
+ ```bash
663
+ vkit scan production_pg
664
+
665
+ # Output:
666
+ # Scanning datasource: production_pg
667
+ #
668
+ # Schema Drift Detected:
669
+ #
670
+ # + Dataset: customers
671
+ # + Field: phone_number (varchar) [PII] ⚠️ NEW - requires policy
672
+ # ~ Field: email (text → varchar) [PII] ✓ Already masked
673
+ #
674
+ # + Dataset: user_sessions
675
+ # + Field: ip_address (inet) [PII] ⚠️ NEW - requires policy
676
+ # + Field: user_agent (text) [INTERNAL] ⚠️ NEW - requires policy
677
+ #
678
+ # Summary: 2 new datasets, 3 new fields, 1 modified field
739
679
  ```
740
680
 
741
- **How it works:**
742
- 1. AI agents receive short-lived session tokens (15 minutes)
743
- 2. All PII fields automatically hashed before results return
744
- 3. Agents can analyze patterns without accessing raw sensitive data
745
- 4. Every query logged with AI agent identity
681
+ **Apply Changes:**
682
+ ```bash
683
+ vkit scan production_pg --mode apply
684
+
685
+ # Output:
686
+ # ✓ Baseline registry updated
687
+ # Next steps:
688
+ # 1. Review new fields: customers.phone_number, user_sessions.ip_address
689
+ # 2. Update policies in Git: config/policies/
690
+ # 3. Build new bundle: vkit policy bundle
691
+ # 4. Deploy bundle: vkit policy deploy
692
+ ```
746
693
 
747
- **CLI Usage:**
694
+ **Scan All Datasources:**
748
695
  ```bash
749
- # AI agent requests data
750
- vkit request \
751
- --datasource production_db \
752
- --role ai_agent \
753
- --aql '{
754
- "source_table": "customer_behavior",
755
- "columns": ["user_id", "email", "purchase_amount"],
756
- "filters": [{"field": "purchase_amount", "operator": "gt", "value": 1000}]
757
- }'
696
+ for ds in $(vkit datasource list --format json | jq -r '.[].id'); do
697
+ echo "Scanning $ds..."
698
+ vkit scan $ds
699
+ done
700
+ ```
701
+
702
+ ---
703
+
704
+ ### Policy Management Commands
705
+
706
+ #### `vkit policy bundle`
707
+
708
+ Compile policies and registry into a deployable bundle.
709
+
710
+ ```bash
711
+ vkit policy bundle [OPTIONS]
712
+ ```
713
+
714
+ **Options:**
715
+ - `--policies_dir` — Path to policies directory (default: `config/policies`)
716
+ - `--registry_dir` — Path to registry files (default: `config`)
717
+ - `--datasources_dir` — Path to datasource configs (default: `config/datasources`)
718
+ - `--out` — Output bundle file (default: `dist/policy_bundle.json`)
719
+ - `--org` — Organization identifier
720
+
721
+ **Example:**
722
+ ```bash
723
+ vkit policy bundle \
724
+ --policies_dir config/policies \
725
+ --registry_dir config \
726
+ --datasources_dir config/datasources \
727
+ --out dist/policy_bundle.json \
728
+ --org acme
729
+ ```
730
+
731
+ #### `vkit policy validate`
732
+
733
+ Validate a policy bundle without deploying.
734
+
735
+ ```bash
736
+ vkit policy validate --bundle <BUNDLE_FILE>
737
+ ```
738
+
739
+ **Example:**
740
+ ```bash
741
+ vkit policy validate --bundle dist/policy_bundle.json
742
+
743
+ # Output:
744
+ # Validating policy bundle...
745
+ # ✓ Schema validation passed
746
+ # ✓ Policy syntax valid
747
+ # ✓ No circular dependencies
748
+ # ✓ All datasources referenced
749
+ # Bundle is valid and ready to deploy
750
+ ```
758
751
 
759
- # Result: email field is hashed
760
- # {
761
- # "user_id": 12345,
762
- # "email": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
763
- # "purchase_amount": 1250.00
764
- # }
752
+ #### `vkit policy deploy`
753
+
754
+ Deploy a policy bundle to the control plane.
755
+
756
+ ```bash
757
+ vkit policy deploy --bundle <BUNDLE_FILE> [OPTIONS]
758
+ ```
759
+
760
+ **Options:**
761
+ - `--bundle` — Path to bundle file (required)
762
+ - `--org` — Organization identifier
763
+ - `--activate` — Immediately activate bundle
764
+ - `--dry-run` — Validate deployment without activating
765
+
766
+ **Examples:**
767
+ ```bash
768
+ # Deploy and activate
769
+ vkit policy deploy \
770
+ --bundle dist/policy_bundle.json \
771
+ --org acme \
772
+ --activate
773
+
774
+ # Dry run (test deployment)
775
+ vkit policy deploy \
776
+ --bundle dist/policy_bundle.json \
777
+ --org acme \
778
+ --dry-run
779
+ ```
780
+
781
+ #### `vkit policy list`
782
+
783
+ List active policy bundles.
784
+
785
+ ```bash
786
+ vkit policy list
787
+ ```
788
+
789
+ #### `vkit policy show`
790
+
791
+ Show details of a specific policy.
792
+
793
+ ```bash
794
+ vkit policy show <POLICY_ID>
765
795
  ```
766
796
 
767
797
  ---
768
798
 
769
- ### 2. **Cross-Region Compliance (GDPR, CCPA, HIPAA)**
770
- Automatically mask or deny access to sensitive fields based on user location and data residency requirements.
799
+ ### Audit Commands
771
800
 
772
- **Problem:**
773
- - GDPR prohibits transferring EU citizen data outside EU
774
- - CCPA requires disclosure of data access by California residents
775
- - HIPAA restricts PHI access based on business need
776
- - Manual compliance is error-prone and doesn't scale
801
+ #### `vkit audit query`
777
802
 
778
- **VaultKit Solution:**
803
+ Search audit logs.
804
+
805
+ ```bash
806
+ vkit audit query [OPTIONS]
807
+ ```
808
+
809
+ **Options:**
810
+ - `--user` — Filter by user email
811
+ - `--action` — Filter by action: `granted`, `denied`, `pending`
812
+ - `--dataset` — Filter by dataset name
813
+ - `--days` — Look back N days (default: 7)
814
+ - `--limit` — Number of results (default: 100)
815
+ - `--format` — Output format: `json`, `table`, `csv`
816
+
817
+ **Examples:**
818
+ ```bash
819
+ # All queries by a user
820
+ vkit audit query --user analyst@company.com --days 30
821
+
822
+ # Denied access attempts
823
+ vkit audit query --action denied --days 7
824
+
825
+ # Access to specific dataset
826
+ vkit audit query --dataset customers --action granted
827
+
828
+ # Export to CSV
829
+ vkit audit query --user analyst@company.com --format csv --output audit_report.csv
830
+ ```
831
+
832
+ #### `vkit audit export`
833
+
834
+ Export audit logs for compliance reporting.
835
+
836
+ ```bash
837
+ vkit audit export --start-date <DATE> --end-date <DATE> [OPTIONS]
838
+ ```
839
+
840
+ **Options:**
841
+ - `--start-date` — Start date (YYYY-MM-DD)
842
+ - `--end-date` — End date (YYYY-MM-DD)
843
+ - `--format` — Output format: `json`, `csv` (default: `csv`)
844
+ - `--output` — Output file path
845
+
846
+ **Example:**
847
+ ```bash
848
+ vkit audit export \
849
+ --start-date 2024-01-01 \
850
+ --end-date 2024-01-31 \
851
+ --format csv \
852
+ --output compliance_reports/january_2024.csv
853
+ ```
854
+
855
+ ---
856
+
857
+ ## ⚙️ Configuration
858
+
859
+ ### Configuration File
860
+
861
+ Create `~/.vkit/config.yaml` for persistent settings:
779
862
 
780
863
  ```yaml
781
- # policies/gdpr_protection.yaml
782
- id: gdpr_cross_region_deny
783
- match:
784
- dataset: eu_customers
785
- fields:
786
- category: pii
787
-
788
- context:
789
- requester_region: US
790
- dataset_region: EU
791
-
792
- action:
793
- deny: true
794
- reason: "Cross-region PII access forbidden due to GDPR Article 44"
795
- audit_flag: compliance_violation
864
+ # VaultKit Control Plane
865
+ api_url: "https://vaultkit.company.com"
866
+
867
+ # Authentication
868
+ auth:
869
+ method: "sso" # or "password"
870
+ sso_provider: "okta"
871
+
872
+ # Default Settings
873
+ defaults:
874
+ datasource: "production_pg"
875
+ environment: "production"
876
+ requester_region: "US"
877
+ clearance_level: "high"
878
+
879
+ # Output Preferences
880
+ output:
881
+ format: "table" # json, table, csv
882
+ color: true
883
+
884
+ # Audit Logging
885
+ audit:
886
+ local_log: true
887
+ log_customers_path: "~/.vkit/audit.log"
888
+
889
+ # Request Settings
890
+ request:
891
+ default_ttl: 3600 # 1 hour
892
+ auto_retry: true
893
+ retry_attempts: 3
894
+ ```
895
+
896
+ ### Environment Variables
897
+
898
+ Override config file with environment variables:
899
+
900
+ ```bash
901
+ # Required
902
+ export VKIT_API_URL="https://vaultkit.company.com"
903
+
904
+ # Optional
905
+ export VKIT_CONFIg_customers_PATH="~/.vkit/config.yaml"
906
+ export VKIT_AUTH_TOKEN="eyJhbGciOiJIUzI1NiIs..."
907
+ export VKIT_DEFAULT_DATASOURCE="production_pg"
908
+ export VKIT_OUTPUT_FORMAT="json"
909
+ ```
910
+
911
+ ### Precedence Order
912
+
913
+ 1. Command-line flags (highest priority)
914
+ 2. Environment variables
915
+ 3. Configuration file
916
+ 4. Built-in defaults (lowest priority)
796
917
 
797
918
  ---
798
- id: ccpa_disclosure_logging
799
- match:
800
- dataset: california_residents
801
919
 
802
- context:
803
- requester_region: ANY
920
+ ## 🔬 Advanced Usage
921
+
922
+ ### Scripting with vkit
923
+
924
+ **Batch Request Processing:**
804
925
 
805
- action:
806
- allow: true
807
- ttl: "1h"
808
- audit_metadata:
809
- ccpa_disclosure: true
810
- data_subject_rights: "User has right to know about this access"
926
+ ```bash
927
+ #!/bin/bash
928
+ # process_requests.sh
929
+
930
+ REQUESTS=(
931
+ '{"source_table":"customers","columns":["id","email"],"limit":100}'
932
+ '{"source_table":"orders","columns":["order_id","total"],"limit":100}'
933
+ '{"source_table":"products","columns":["product_id","name"],"limit":100}'
934
+ )
935
+
936
+ for aql in "${REQUESTS[@]}"; do
937
+ echo "Processing: $aql"
938
+ vkit request --aql "$aql" --format json | jq -r '.grant_id' >> grants.txt
939
+ done
940
+
941
+ echo "Generated grants:"
942
+ cat grants.txt
811
943
  ```
812
944
 
813
- **How it works:**
814
- - US-based analysts querying EU customer data are automatically denied
815
- - Policy engine evaluates both requester and dataset regions
816
- - Audit log captures denied attempts with compliance flags
817
- - CCPA-flagged queries generate disclosure reports
945
+ **Automated Approvals (for testing):**
818
946
 
819
- **Multi-Region Access Pattern:**
820
947
  ```bash
821
- # US analyst attempts to query EU data
822
- vkit request --datasource eu_customers_db --aql '{...}'
948
+ #!/bin/bash
949
+ # auto_approve_pending.sh
950
+
951
+ PENDING=$(vkit approval:list --state pending --format json)
823
952
 
824
- # Response:
825
- # Access Denied
826
- # Reason: Cross-region PII access forbidden due to GDPR Article 44
827
- # Policy: gdpr_cross_region_deny
828
- # Audit ID: audit_20240115_abc123
953
+ echo "$PENDING" | jq -r '.[].id' | while read -r request_id; do
954
+ echo "Approving request: $request_id"
955
+ vkit approval:approve "$request_id" --ttl 3600 --notes "Auto-approved for testing"
956
+ done
957
+ ```
958
+
959
+ ### Integration with CI/CD
960
+
961
+ **GitHub Actions Example:**
962
+
963
+ ```yaml
964
+ name: Deploy VaultKit Policies
965
+
966
+ on:
967
+ push:
968
+ branches: [main]
969
+ paths:
970
+ - 'config/policies/**'
971
+ - 'config/registry.yaml'
972
+
973
+ jobs:
974
+ deploy:
975
+ runs-on: ubuntu-latest
976
+ steps:
977
+ - uses: actions/checkout@v3
978
+
979
+ - name: Install vkit CLI
980
+ run: gem install vaultkitcli
981
+
982
+ - name: Authenticate
983
+ run: |
984
+ vkit login \
985
+ --endpoint ${{ secrets.VKIT_API_URL }} \
986
+ --email ${{ secrets.VKIT_BOT_EMAIL }} \
987
+ --password ${{ secrets.VKIT_BOT_PASSWORD }}
988
+
989
+ - name: Build Policy Bundle
990
+ run: |
991
+ vkit policy bundle \
992
+ --policies_dir config/policies \
993
+ --registry_dir config \
994
+ --out policy_bundle.json \
995
+ --org ${{ secrets.ORg_customers_ID }}
996
+
997
+ - name: Validate Bundle
998
+ run: vkit policy validate --bundle policy_bundle.json
999
+
1000
+ - name: Deploy Bundle
1001
+ run: |
1002
+ vkit policy deploy \
1003
+ --bundle policy_bundle.json \
1004
+ --org ${{ secrets.ORg_customers_ID }} \
1005
+ --activate
1006
+ ```
1007
+
1008
+ ### Using jq for Advanced Filtering
1009
+
1010
+ **Extract specific fields:**
1011
+ ```bash
1012
+ vkit requests list --format json | jq '.[] | select(.state == "approved") | {id, dataset, requester}'
1013
+ ```
1014
+
1015
+ **Count requests by state:**
1016
+ ```bash
1017
+ vkit requests list --format json | jq 'group_by(.state) | map({state: .[0].state, count: length})'
1018
+ ```
1019
+
1020
+ **Find high-value transactions:**
1021
+ ```bash
1022
+ vkit fetch --grant g_customers_abc123 --format json | jq '.data[] | select(.amount > 10000)'
829
1023
  ```
830
1024
 
831
1025
  ---
832
1026
 
833
- ### 3. **Just-In-Time Access (Break-Glass)**
834
- Developers can request temporary elevated access to production data with automatic approval workflows and audit trails.
1027
+ ## 🐛 Troubleshooting
1028
+
1029
+ ### Common Issues
1030
+
1031
+ #### Authentication Failed
835
1032
 
836
1033
  **Problem:**
837
- - Engineers need emergency production access for incident resolution
838
- - Standing privileges violate principle of least privilege
839
- - Manual approval processes slow incident response
840
- - Audit trails are incomplete or manual
1034
+ ```
1035
+ Error: Authentication failed (401 Unauthorized)
1036
+ ```
841
1037
 
842
- **VaultKit Solution:**
1038
+ **Solutions:**
1039
+ ```bash
1040
+ # 1. Check endpoint
1041
+ echo $VKIT_API_URL
843
1042
 
844
- ```yaml
845
- # policies/financial_requires_approval.yaml
846
- id: financial_break_glass
847
- match:
848
- fields:
849
- category: financial
850
-
851
- context:
852
- environment: production
853
- requester_role: engineer
854
-
855
- action:
856
- require_approval: true
857
- approver_role: finance_manager
858
- approval_metadata_required:
859
- - incident_ticket
860
- - business_justification
861
- ttl: "1h"
862
- reason: "Financial data access requires approval"
863
- auto_revoke: true
864
- ```
865
-
866
- **CLI Usage:**
867
- ```bash
868
- # Engineer requests emergency access
869
- vkit request \
870
- --datasource production_db \
871
- --approval-required \
872
- --reason "Investigating payment processing failure - Ticket INC-5432" \
873
- --metadata '{"incident_ticket": "INC-5432", "business_justification": "Payment processor reporting transaction mismatch"}' \
874
- --duration "2h" \
875
- --aql '{
876
- "source_table": "transactions",
877
- "columns": ["transaction_id", "amount", "status"],
878
- "filters": [{"field": "status", "operator": "eq", "value": "failed"}]
879
- }'
1043
+ # 2. Re-authenticate
1044
+ vkit logout
1045
+ vkit login --endpoint http://localhost:3000 --email you@company.com
880
1046
 
881
- # Response:
882
- # ⏳ Approval Required
883
- # Request ID: req_20240115_xyz789
884
- # Status: PENDING
885
- # Approver: finance_manager
886
- # Expires: 2024-01-15 18:30 UTC
1047
+ # 3. Verify credentials
1048
+ vkit whoami
887
1049
  ```
888
1050
 
889
- **Approval Workflow:**
1051
+ #### Connection Refused
1052
+
1053
+ **Problem:**
1054
+ ```
1055
+ Error: Connection refused - connect(2) for "localhost" port 3000
1056
+ ```
1057
+
1058
+ **Solutions:**
890
1059
  ```bash
891
- # Finance manager reviews request
892
- vkit approval list --pending
1060
+ # 1. Check if control plane is running
1061
+ curl http://localhost:3000/health
893
1062
 
894
- # Approve with notes
895
- vkit approval grant \
896
- --request-id req_20240115_xyz789 \
897
- --notes "Approved for incident resolution. Access limited to failed transactions only."
1063
+ # 2. Verify Docker containers
1064
+ cd infra
1065
+ docker compose ps
898
1066
 
899
- # Engineer notified and can fetch data
900
- vkit fetch --grant grant_approved_abc123
1067
+ # 3. Start services
1068
+ docker compose up
901
1069
  ```
902
1070
 
903
- **Audit Trail Output:**
904
- ```json
905
- {
906
- "request_id": "req_20240115_xyz789",
907
- "requester": "engineer@company.com",
908
- "requester_role": "engineer",
909
- "requested_at": "2024-01-15T16:00:00Z",
910
- "approved_by": "finance.manager@company.com",
911
- "approved_at": "2024-01-15T16:05:32Z",
912
- "approval_reason": "Approved for incident resolution",
913
- "access_granted": "2024-01-15T16:05:32Z",
914
- "access_expires": "2024-01-15T18:05:32Z",
915
- "queries_executed": 3,
916
- "rows_accessed": 147,
917
- "incident_ticket": "INC-5432",
918
- "auto_revoked": true
919
- }
1071
+ #### Grant Expired
1072
+
1073
+ **Problem:**
1074
+ ```
1075
+ Error: Grant has expired
1076
+ Grant ID: g_customers_abc123xyz
1077
+ Expired at: 2024-01-15 12:00:00 UTC
920
1078
  ```
921
1079
 
922
- ---
1080
+ **Solutions:**
1081
+ ```bash
1082
+ # 1. Request new grant
1083
+ vkit request --aql '{"source_table":"customers",...}'
1084
+
1085
+ # 2. Request longer TTL (if approval required)
1086
+ vkit request --aql '{...}' --ttl 7200
1087
+ ```
923
1088
 
924
- ### 4. **Time-Restricted Access Windows**
925
- Enforce access controls based on business hours, maintenance windows, or compliance requirements.
1089
+ #### Policy Denial
926
1090
 
927
1091
  **Problem:**
928
- - Audit logs should only be accessible during business hours
929
- - Production data access outside work hours indicates potential misuse
930
- - Compliance requires time-based controls for certain datasets
1092
+ ```
1093
+ Request denied
1094
+ Reason: Cross-region PII access forbidden
1095
+ Policy: gdpr_protection
1096
+ ```
931
1097
 
932
- **VaultKit Solution:**
1098
+ **Solutions:**
1099
+ 1. Review policy restrictions
1100
+ 2. Request approval if available
1101
+ 3. Modify query to exclude restricted fields
1102
+ 4. Contact admin to update policies
933
1103
 
934
- ```yaml
935
- # policies/time_restricted_access.yaml
936
- id: business_hours_only
937
- match:
938
- dataset: audit_logs
939
- environment: production
940
-
941
- context:
942
- time:
943
- start: "08:00"
944
- end: "18:00"
945
- timezone: "America/Toronto"
946
- days: ["monday", "tuesday", "wednesday", "thursday", "friday"]
947
-
948
- action:
949
- allow: true
950
- reason: "Access allowed during business hours"
951
- ttl: "1h"
1104
+ ### Debug Mode
1105
+
1106
+ Enable verbose logging:
1107
+
1108
+ ```bash
1109
+ # Set debug environment variable
1110
+ export VKIT_DEBUG=true
1111
+
1112
+ # Run command
1113
+ vkit request --aql '{...}'
1114
+
1115
+ # Output includes:
1116
+ # - HTTP request/response details
1117
+ # - Policy evaluation trace
1118
+ # - Timing information
1119
+ ```
1120
+
1121
+ ### Health Check
1122
+
1123
+ Verify CLI and control plane connectivity:
1124
+
1125
+ ```bash
1126
+ # Check CLI version
1127
+ vkit --version
1128
+
1129
+ # Check control plane health
1130
+ curl $VKIT_API_URL/health
1131
+
1132
+ # Test authentication
1133
+ vkit whoami
1134
+ ```
952
1135
 
953
1136
  ---
954
- id: after_hours_deny
955
- match:
956
- dataset: audit_logs
957
- environment: production
958
-
959
- context:
960
- time:
961
- outside_window
1137
+
1138
+ ## 🤝 Contributing
1139
+
1140
+ We welcome contributions to the VaultKit CLI!
1141
+
1142
+ ### Development Setup
1143
+
1144
+ ```bash
1145
+ # Clone repository
1146
+ git clone https://github.com/yourorg/vaultkit-cli.git
1147
+ cd vaultkit-cli
1148
+
1149
+ # Install dependencies
1150
+ bundle install
1151
+
1152
+ # Run tests
1153
+ bundle exec rspec
1154
+
1155
+ # Run CLI locally
1156
+ bundle exec bin/vkit --help
1157
+ ```
1158
+
1159
+ ### Running Tests
1160
+
1161
+ ```bash
1162
+ # Unit tests
1163
+ bundle exec rspec spec/unit
1164
+
1165
+ # Integration tests (requires running control plane)
1166
+ bundle exec rspec spec/integration
1167
+
1168
+ # All tests
1169
+ bundle exec rspec
1170
+ ```
1171
+
1172
+ ### Code Style
1173
+
1174
+ ```bash
1175
+ # Run RuboCop
1176
+ bundle exec rubocop
1177
+
1178
+ # Auto-fix issues
1179
+ bundle exec rubocop -a
1180
+ ```
1181
+
1182
+ ---
1183
+
1184
+ ## 📚 Additional Resources
1185
+
1186
+ - **Main Repository**: [github.com/yourorg/vaultkit](https://github.com/yourorg/vaultkit)
1187
+ - **Documentation**: [docs.vaultkit.io](https://docs.vaultkit.io)
1188
+ - **AQL Specification**: [docs.vaultkit.io/aql](https://docs.vaultkit.io/aql)
1189
+ - **Policy Reference**: [docs.vaultkit.io/policies](https://docs.vaultkit.io/policies)
1190
+ - **API Documentation**: [docs.vaultkit.io/api](https://docs.vaultkit.io/api)
1191
+
1192
+ ---
1193
+
1194
+ ## 📄 License
1195
+
1196
+ VaultKit CLI is licensed under the Apache License 2.0. See [LICENSE](LICENSE) for details.
1197
+
1198
+ ---
1199
+
1200
+ ## 💬 Support
1201
+
1202
+ - **Issues**: [github.com/yourorg/vaultkit-cli/issues](https://github.com/yourorg/vaultkit-cli/issues)
1203
+ - **Discussions**: [github.com/yourorg/vaultkit-cli/discussions](https://github.com/yourorg/vaultkit-cli/discussions)
1204
+ - **Email**: support@vaultkit.io
1205
+ - **Slack**: [vaultkit.slack.com](https://vaultkit.slack.com)
1206
+
1207
+ ---
1208
+
1209
+ **Built with ❤️ by the VaultKit team**