rust-kgdb 0.6.81 → 0.6.82
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +281 -0
- package/examples/rpc-catalog-dprod-demo.js +339 -0
- package/examples/rpc-federation-sql-demo.js +273 -0
- package/examples/rpc-virtual-tables-demo.js +268 -0
- package/hypermind-agent.js +626 -0
- package/index.d.ts +304 -0
- package/index.js +9 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -8,6 +8,33 @@
|
|
|
8
8
|
|
|
9
9
|
---
|
|
10
10
|
|
|
11
|
+
## What's New in v0.7.0
|
|
12
|
+
|
|
13
|
+
| Feature | Description | Performance |
|
|
14
|
+
|---------|-------------|-------------|
|
|
15
|
+
| **HyperFederate** | Cross-database SQL: KGDB + Snowflake + BigQuery | Single query, 890ms 3-way federation |
|
|
16
|
+
| **RpcFederationProxy** | WASM RPC proxy for federated queries | 7 UDFs + 9 Table Functions |
|
|
17
|
+
| **Virtual Tables** | Session-bound query materialization | No ETL, real-time results |
|
|
18
|
+
| **DCAT DPROD Catalog** | W3C-aligned data product registry | Self-describing RDF storage |
|
|
19
|
+
| **Federation ProofDAG** | Full provenance for federated results | SHA-256 audit trail |
|
|
20
|
+
|
|
21
|
+
```javascript
|
|
22
|
+
const { GraphDB, RpcFederationProxy, FEDERATION_TOOLS } = require('rust-kgdb')
|
|
23
|
+
|
|
24
|
+
// Query across KGDB + Snowflake + BigQuery in single SQL
|
|
25
|
+
const federation = new RpcFederationProxy({ endpoint: 'http://localhost:30180' })
|
|
26
|
+
const result = await federation.query(`
|
|
27
|
+
SELECT kg.*, sf.C_NAME, bq.name_popularity
|
|
28
|
+
FROM graph_search('SELECT ?person WHERE { ?person a :Customer }') kg
|
|
29
|
+
JOIN snowflake.CUSTOMER sf ON kg.custKey = sf.C_CUSTKEY
|
|
30
|
+
LEFT JOIN bigquery.usa_names bq ON sf.C_NAME = bq.name
|
|
31
|
+
`)
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
*See [HyperFederate: Cross-Database Federation](#hyperfederate-cross-database-federation) for complete documentation.*
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
11
38
|
## What's New in v0.6.79
|
|
12
39
|
|
|
13
40
|
| Feature | Description | Performance |
|
|
@@ -1579,11 +1606,265 @@ const agent = new AgentBuilder('scoped-agent')
|
|
|
1579
1606
|
| **Joins** | WCOJ | Worst-case optimal join algorithm |
|
|
1580
1607
|
| **Distribution** | HDRF | Streaming graph partitioning |
|
|
1581
1608
|
| **Distribution** | Raft | Consensus for coordination |
|
|
1609
|
+
| **Federation** | HyperFederate | Cross-database SQL: KGDB + Snowflake + BigQuery |
|
|
1610
|
+
| **Federation** | Virtual Tables | Session-bound query materialization |
|
|
1611
|
+
| **Federation** | DCAT Catalog | W3C DPROD data product registry |
|
|
1582
1612
|
| **Mobile** | iOS/Android | Swift and Kotlin bindings via UniFFI |
|
|
1583
1613
|
| **Storage** | InMemory/RocksDB/LMDB | Three backend options |
|
|
1584
1614
|
|
|
1585
1615
|
---
|
|
1586
1616
|
|
|
1617
|
+
## HyperFederate: Cross-Database Federation
|
|
1618
|
+
|
|
1619
|
+
### The Real Problem: Your Knowledge Lives Everywhere
|
|
1620
|
+
|
|
1621
|
+
Here's what actually happens in enterprise AI projects:
|
|
1622
|
+
|
|
1623
|
+
A fraud analyst asks: *"Show me high-risk customers with large account balances and unusual name patterns."*
|
|
1624
|
+
|
|
1625
|
+
To answer this, they need:
|
|
1626
|
+
- **Risk scores** from the Knowledge Graph (semantic relationships, fraud patterns)
|
|
1627
|
+
- **Account balances** from Snowflake (transaction history, customer master)
|
|
1628
|
+
- **Name demographics** from BigQuery (population statistics, anomaly detection)
|
|
1629
|
+
|
|
1630
|
+
Today's reality? Three separate queries. Manual data exports. Excel joins. Python scripts. Data engineers on standby. Days of work for a single question.
|
|
1631
|
+
|
|
1632
|
+
**This is insane.**
|
|
1633
|
+
|
|
1634
|
+
Your knowledge isn't siloed because you want it to be. It's siloed because no tool could query across systems... until now.
|
|
1635
|
+
|
|
1636
|
+
### One Query. Three Sources. Real Answers.
|
|
1637
|
+
|
|
1638
|
+
| Query Type | Before (Painful) | With HyperFederate |
|
|
1639
|
+
|------------|------------------|---------------------|
|
|
1640
|
+
| **KG Risk + Snowflake Accounts** | 2 queries + Python join | `JOIN snowflake.CUSTOMER ON kg.custKey = sf.C_CUSTKEY` |
|
|
1641
|
+
| **Snowflake + BigQuery Demographics** | ETL pipeline, 4-6 hours | `LEFT JOIN bigquery.usa_names ON sf.C_NAME = bq.name` |
|
|
1642
|
+
| **Three-Way: KG + SF + BQ** | "Not possible without data warehouse" | **Single SQL statement, 890ms** |
|
|
1643
|
+
|
|
1644
|
+
```sql
|
|
1645
|
+
-- The query that would take days... now takes 890ms
|
|
1646
|
+
SELECT
|
|
1647
|
+
kg.person AS entity,
|
|
1648
|
+
kg.riskScore,
|
|
1649
|
+
entity_type(kg.person) AS types, -- Semantic UDF
|
|
1650
|
+
similar_to(kg.person, 0.6) AS related, -- AI-powered similarity
|
|
1651
|
+
sf.C_NAME AS customer_name,
|
|
1652
|
+
sf.C_ACCTBAL AS account_balance,
|
|
1653
|
+
bq.name AS popular_name,
|
|
1654
|
+
bq.number AS name_popularity
|
|
1655
|
+
FROM graph_search('SELECT ?person ?riskScore WHERE { ?person :riskScore ?riskScore }') kg
|
|
1656
|
+
JOIN snowflake_tpch.CUSTOMER sf ON CAST(kg.custKey AS INT) = sf.C_CUSTKEY
|
|
1657
|
+
LEFT JOIN bigquery_public.usa_names bq ON LOWER(sf.C_NAME) = LOWER(bq.name)
|
|
1658
|
+
WHERE kg.riskScore > 0.7
|
|
1659
|
+
LIMIT 10
|
|
1660
|
+
```
|
|
1661
|
+
|
|
1662
|
+
**The analyst gets their answer in under a second.** No data engineers. No ETL. No waiting.
|
|
1663
|
+
|
|
1664
|
+
### How It Works: Heavy Lifting in Rust Core
|
|
1665
|
+
|
|
1666
|
+
The TypeScript SDK is intentionally thin. A thin RPC proxy. All the hard work happens in Rust:
|
|
1667
|
+
|
|
1668
|
+
```
|
|
1669
|
+
┌─────────────────────────────────────────────────────────────────────────────────┐
|
|
1670
|
+
│ TypeScript SDK (Thin RPC Proxy) │
|
|
1671
|
+
│ RpcFederationProxy: query(), createVirtualTable(), listCatalog(), ... │
|
|
1672
|
+
└─────────────────────────────────────────────────────────────────────────────────┘
|
|
1673
|
+
│ HTTP/RPC
|
|
1674
|
+
▼
|
|
1675
|
+
┌─────────────────────────────────────────────────────────────────────────────────┐
|
|
1676
|
+
│ Rust HyperFederate Core │
|
|
1677
|
+
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
1678
|
+
│ │ Apache Arrow │ │ Memory │ │ HDRF │ │ Category │ │
|
|
1679
|
+
│ │ / Flight │ │ Acceleration │ │ Partitioner │ │ Theory │ │
|
|
1680
|
+
│ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
1681
|
+
│ │
|
|
1682
|
+
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
|
1683
|
+
│ │ Connector Registry (5+ Sources) │ │
|
|
1684
|
+
│ │ KGDB (graph_search) │ Snowflake │ BigQuery │ PostgreSQL │ MySQL │ │
|
|
1685
|
+
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
|
1686
|
+
└─────────────────────────────────────────────────────────────────────────────────┘
|
|
1687
|
+
```
|
|
1688
|
+
|
|
1689
|
+
- **Apache Arrow/Flight**: High-performance columnar SQL engine (Rust)
|
|
1690
|
+
- **Memory Acceleration**: Zero-copy data transfer for sub-second queries
|
|
1691
|
+
- **HDRF**: Subject-anchored partitioning for distributed execution
|
|
1692
|
+
- **Category Theory**: Tools as typed morphisms with provable correctness
|
|
1693
|
+
|
|
1694
|
+
### Why This Matters
|
|
1695
|
+
|
|
1696
|
+
| Capability | rust-kgdb + HyperFederate | Competitors |
|
|
1697
|
+
|------------|---------------------------|-------------|
|
|
1698
|
+
| **Cross-DB SQL** | ✅ JOIN across 5+ sources | ❌ Single source only |
|
|
1699
|
+
| **KG Integration** | ✅ SPARQL in SQL | ❌ Separate systems |
|
|
1700
|
+
| **Semantic UDFs** | ✅ 7 AI-powered functions | ❌ None |
|
|
1701
|
+
| **Table Functions** | ✅ 9 graph analytics | ❌ Basic aggregates |
|
|
1702
|
+
| **Virtual Tables** | ✅ Session-bound materialization | ❌ ETL required |
|
|
1703
|
+
| **Data Catalog** | ✅ DCAT DPROD ontology | ❌ Proprietary |
|
|
1704
|
+
| **Proof/Lineage** | ✅ Full provenance (W3C PROV) | ❌ None |
|
|
1705
|
+
|
|
1706
|
+
### Using RpcFederationProxy
|
|
1707
|
+
|
|
1708
|
+
```javascript
|
|
1709
|
+
const { RpcFederationProxy, ProofDAG } = require('rust-kgdb')
|
|
1710
|
+
|
|
1711
|
+
const federation = new RpcFederationProxy({
|
|
1712
|
+
endpoint: 'http://localhost:30180',
|
|
1713
|
+
identityId: 'risk-analyst-001'
|
|
1714
|
+
})
|
|
1715
|
+
|
|
1716
|
+
// Query across KGDB + Snowflake + BigQuery in single SQL
|
|
1717
|
+
const result = await federation.query(`
|
|
1718
|
+
WITH kg_risk AS (
|
|
1719
|
+
SELECT * FROM graph_search('
|
|
1720
|
+
PREFIX finance: <https://gonnect.ai/domains/finance#>
|
|
1721
|
+
SELECT ?person ?riskScore WHERE {
|
|
1722
|
+
?person finance:riskScore ?riskScore .
|
|
1723
|
+
FILTER(?riskScore > 0.7)
|
|
1724
|
+
}
|
|
1725
|
+
')
|
|
1726
|
+
)
|
|
1727
|
+
SELECT
|
|
1728
|
+
kg.person AS entity,
|
|
1729
|
+
kg.riskScore,
|
|
1730
|
+
-- Semantic UDFs on KG entities
|
|
1731
|
+
entity_type(kg.person) AS types,
|
|
1732
|
+
similar_to(kg.person, 0.6) AS similar_entities,
|
|
1733
|
+
-- Snowflake customer data
|
|
1734
|
+
sf.C_NAME AS customer_name,
|
|
1735
|
+
sf.C_ACCTBAL AS account_balance,
|
|
1736
|
+
-- BigQuery demographics
|
|
1737
|
+
bq.name AS popular_name,
|
|
1738
|
+
bq.number AS name_popularity
|
|
1739
|
+
FROM kg_risk kg
|
|
1740
|
+
JOIN snowflake_tpch.CUSTOMER sf ON CAST(kg.custKey AS INT) = sf.C_CUSTKEY
|
|
1741
|
+
LEFT JOIN bigquery_public.usa_names bq ON LOWER(sf.C_NAME) = LOWER(bq.name)
|
|
1742
|
+
LIMIT 10
|
|
1743
|
+
`)
|
|
1744
|
+
|
|
1745
|
+
console.log(`Returned ${result.rowCount} rows in ${result.duration}ms`)
|
|
1746
|
+
console.log(`Sources: ${result.metadata.sources.join(', ')}`)
|
|
1747
|
+
```
|
|
1748
|
+
|
|
1749
|
+
### Semantic UDFs (7 AI-Powered Functions)
|
|
1750
|
+
|
|
1751
|
+
| UDF | Signature | Description |
|
|
1752
|
+
|-----|-----------|-------------|
|
|
1753
|
+
| `similar_to` | `(entity, threshold)` | Find semantically similar entities via RDF2Vec |
|
|
1754
|
+
| `text_search` | `(query, limit)` | Semantic text search |
|
|
1755
|
+
| `neighbors` | `(entity, hops)` | N-hop graph traversal |
|
|
1756
|
+
| `graph_pattern` | `(s, p, o)` | Triple pattern matching |
|
|
1757
|
+
| `sparql_query` | `(sparql)` | Inline SPARQL execution |
|
|
1758
|
+
| `entity_type` | `(entity)` | Get RDF types |
|
|
1759
|
+
| `entity_properties` | `(entity)` | Get all properties |
|
|
1760
|
+
|
|
1761
|
+
### Table Functions (9 Graph Analytics)
|
|
1762
|
+
|
|
1763
|
+
| Function | Description |
|
|
1764
|
+
|----------|-------------|
|
|
1765
|
+
| `graph_search(sparql)` | SPARQL → SQL bridge |
|
|
1766
|
+
| `vector_search(text, k, threshold)` | Semantic similarity search |
|
|
1767
|
+
| `pagerank(sparql, damping, iterations)` | PageRank centrality |
|
|
1768
|
+
| `connected_components(sparql)` | Community detection |
|
|
1769
|
+
| `shortest_paths(src, dst, max_hops)` | Path finding |
|
|
1770
|
+
| `triangle_count(sparql)` | Graph density measure |
|
|
1771
|
+
| `label_propagation(sparql, iterations)` | Community detection |
|
|
1772
|
+
| `datalog_reason(rules)` | Datalog inference |
|
|
1773
|
+
| `motif_search(pattern)` | Graph pattern matching |
|
|
1774
|
+
|
|
1775
|
+
### Virtual Tables (Session-Bound Materialization)
|
|
1776
|
+
|
|
1777
|
+
```javascript
|
|
1778
|
+
// Create virtual table from federation query
|
|
1779
|
+
const vt = await federation.createVirtualTable('high_risk_customers', `
|
|
1780
|
+
SELECT kg.*, sf.C_ACCTBAL
|
|
1781
|
+
FROM graph_search('SELECT ?person ?riskScore WHERE {...}') kg
|
|
1782
|
+
JOIN snowflake.CUSTOMER sf ON ...
|
|
1783
|
+
WHERE kg.riskScore > 0.8
|
|
1784
|
+
`, {
|
|
1785
|
+
refreshPolicy: 'on_demand', // or 'ttl', 'on_source_change'
|
|
1786
|
+
ttlSeconds: 3600,
|
|
1787
|
+
sharedWith: ['risk-analyst-002'],
|
|
1788
|
+
sharedWithGroups: ['team-risk-analytics']
|
|
1789
|
+
})
|
|
1790
|
+
|
|
1791
|
+
// Query without re-execution (materialized)
|
|
1792
|
+
const filtered = await federation.queryVirtualTable(
|
|
1793
|
+
'high_risk_customers',
|
|
1794
|
+
'C_ACCTBAL > 100000'
|
|
1795
|
+
)
|
|
1796
|
+
```
|
|
1797
|
+
|
|
1798
|
+
**Virtual Table Features**:
|
|
1799
|
+
- Session isolation (each user sees only their tables)
|
|
1800
|
+
- Access control via `sharedWith` and `sharedWithGroups`
|
|
1801
|
+
- Stored as RDF triples in KGDB (self-describing)
|
|
1802
|
+
- Queryable via SPARQL for metadata
|
|
1803
|
+
|
|
1804
|
+
### DCAT DPROD Catalog
|
|
1805
|
+
|
|
1806
|
+
```javascript
|
|
1807
|
+
// Register data product in catalog
|
|
1808
|
+
const product = await federation.registerDataProduct({
|
|
1809
|
+
name: 'High Risk Customer Analysis',
|
|
1810
|
+
description: 'Cross-domain risk scoring combining KG + transactional data',
|
|
1811
|
+
sources: ['kgdb', 'snowflake', 'bigquery'],
|
|
1812
|
+
outputPort: '/api/v1/products/high-risk/query',
|
|
1813
|
+
schema: {
|
|
1814
|
+
columns: [
|
|
1815
|
+
{ name: 'entity', type: 'STRING' },
|
|
1816
|
+
{ name: 'riskScore', type: 'FLOAT64' },
|
|
1817
|
+
{ name: 'accountBalance', type: 'DECIMAL(15,2)' }
|
|
1818
|
+
]
|
|
1819
|
+
},
|
|
1820
|
+
quality: {
|
|
1821
|
+
completeness: 0.98,
|
|
1822
|
+
accuracy: 0.95,
|
|
1823
|
+
timeliness: 0.99
|
|
1824
|
+
},
|
|
1825
|
+
owner: 'team-risk-analytics'
|
|
1826
|
+
})
|
|
1827
|
+
|
|
1828
|
+
// List catalog entries
|
|
1829
|
+
const catalog = await federation.listCatalog({ owner: 'team-risk-analytics' })
|
|
1830
|
+
```
|
|
1831
|
+
|
|
1832
|
+
### ProofDAG with Federation Evidence
|
|
1833
|
+
|
|
1834
|
+
```javascript
|
|
1835
|
+
const proof = new ProofDAG('High-risk customers identified across 3 data sources')
|
|
1836
|
+
|
|
1837
|
+
// Add federation evidence to the proof
|
|
1838
|
+
const fedNode = proof.addFederationEvidence(
|
|
1839
|
+
proof.rootId,
|
|
1840
|
+
threeWayQuery, // SQL query
|
|
1841
|
+
['kgdb', 'snowflake', 'bigquery'], // sources
|
|
1842
|
+
42, // rowCount
|
|
1843
|
+
890, // duration (ms)
|
|
1844
|
+
{ planHash: 'abc123', cached: false }
|
|
1845
|
+
)
|
|
1846
|
+
|
|
1847
|
+
console.log(`Proof hash: ${proof.computeHash()}`) // SHA-256 audit trail
|
|
1848
|
+
console.log(`Verification: ${JSON.stringify(proof.verify())}`)
|
|
1849
|
+
```
|
|
1850
|
+
|
|
1851
|
+
### Category Theory Foundation
|
|
1852
|
+
|
|
1853
|
+
HyperFederate tools are typed morphisms following category theory:
|
|
1854
|
+
|
|
1855
|
+
```javascript
|
|
1856
|
+
const { FEDERATION_TOOLS } = require('rust-kgdb')
|
|
1857
|
+
|
|
1858
|
+
// Each tool has Input → Output type signature
|
|
1859
|
+
console.log(FEDERATION_TOOLS['federation.sql.query'])
|
|
1860
|
+
// { input: 'FederatedQuery', output: 'RecordBatch', domain: 'federation' }
|
|
1861
|
+
|
|
1862
|
+
console.log(FEDERATION_TOOLS['federation.udf.call'])
|
|
1863
|
+
// { input: 'UdfCall', output: 'UdfResult', udfs: ['similar_to', 'neighbors', ...] }
|
|
1864
|
+
```
|
|
1865
|
+
|
|
1866
|
+
---
|
|
1867
|
+
|
|
1587
1868
|
## Installation
|
|
1588
1869
|
|
|
1589
1870
|
```bash
|
|
@@ -0,0 +1,339 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* ================================================================================
|
|
4
|
+
* RpcFederationProxy: DCAT DPROD Catalog Demo
|
|
5
|
+
* Enterprise Data Product Registry
|
|
6
|
+
* ================================================================================
|
|
7
|
+
*
|
|
8
|
+
* HyperFederate implements a DCAT DPROD-compliant data product catalog:
|
|
9
|
+
* - Register data products with full metadata
|
|
10
|
+
* - Track sources, schemas, quality metrics, lineage
|
|
11
|
+
* - Discovery via catalog queries
|
|
12
|
+
* - Governance workflows via DPROD ontology
|
|
13
|
+
*
|
|
14
|
+
* DCAT DPROD Ontology (W3C aligned):
|
|
15
|
+
* - dprod:DataProduct - Curated data asset
|
|
16
|
+
* - dprod:DataCatalog - Discovery registry
|
|
17
|
+
* - dprod:InputPort / dprod:OutputPort - Data endpoints
|
|
18
|
+
* - dprod:Schema - Structural metadata
|
|
19
|
+
* - dprod:DataQuality - Quality metrics
|
|
20
|
+
* - dprod:DataLineage - W3C PROV provenance
|
|
21
|
+
*
|
|
22
|
+
* Architecture:
|
|
23
|
+
* - Catalog stored in KGDB as RDF triples (ontology-native)
|
|
24
|
+
* - Heavy lifting in Rust core
|
|
25
|
+
* - TypeScript SDK provides thin RPC proxy layer
|
|
26
|
+
*
|
|
27
|
+
* Run: node examples/rpc-catalog-dprod-demo.js
|
|
28
|
+
*
|
|
29
|
+
* @requires HyperFederate server running at http://localhost:30180
|
|
30
|
+
*/
|
|
31
|
+
|
|
32
|
+
const {
|
|
33
|
+
RpcFederationProxy,
|
|
34
|
+
ProofDAG
|
|
35
|
+
} = require('../index.js')
|
|
36
|
+
|
|
37
|
+
// ================================================================================
|
|
38
|
+
// DEMO: DCAT DPROD Catalog
|
|
39
|
+
// ================================================================================
|
|
40
|
+
|
|
41
|
+
async function runDemo() {
|
|
42
|
+
console.log('╔══════════════════════════════════════════════════════════════════════╗')
|
|
43
|
+
console.log('║ RpcFederationProxy: DCAT DPROD Catalog Demo ║')
|
|
44
|
+
console.log('║ Enterprise Data Product Registry ║')
|
|
45
|
+
console.log('╚══════════════════════════════════════════════════════════════════════╝\n')
|
|
46
|
+
|
|
47
|
+
// Create RpcFederationProxy
|
|
48
|
+
const federation = new RpcFederationProxy({
|
|
49
|
+
endpoint: process.env.HYPERFEDERATE_URL || 'http://localhost:30180',
|
|
50
|
+
identityId: 'data-product-owner-001'
|
|
51
|
+
})
|
|
52
|
+
|
|
53
|
+
console.log(`Federation Endpoint: ${federation.endpoint}`)
|
|
54
|
+
console.log(`Identity ID: ${federation.identityId}`)
|
|
55
|
+
console.log()
|
|
56
|
+
|
|
57
|
+
// ============================================================================
|
|
58
|
+
// 1. Register Data Products
|
|
59
|
+
// ============================================================================
|
|
60
|
+
console.log('1. Register Data Products in Catalog')
|
|
61
|
+
console.log('═'.repeat(70))
|
|
62
|
+
|
|
63
|
+
const dataProducts = [
|
|
64
|
+
{
|
|
65
|
+
name: 'High Risk Customer Analysis',
|
|
66
|
+
description: 'Cross-domain risk scoring combining Knowledge Graph risk assessments with Snowflake transactional data and BigQuery demographics',
|
|
67
|
+
sources: ['kgdb', 'snowflake', 'bigquery'],
|
|
68
|
+
outputPort: '/api/v1/products/high-risk-customers/query',
|
|
69
|
+
schema: {
|
|
70
|
+
columns: [
|
|
71
|
+
{ name: 'kg_entity', type: 'STRING' },
|
|
72
|
+
{ name: 'risk_score', type: 'FLOAT64' },
|
|
73
|
+
{ name: 'entity_types', type: 'ARRAY<STRING>' },
|
|
74
|
+
{ name: 'customer_name', type: 'STRING' },
|
|
75
|
+
{ name: 'account_balance', type: 'DECIMAL(15,2)' },
|
|
76
|
+
{ name: 'name_popularity', type: 'INT64' }
|
|
77
|
+
]
|
|
78
|
+
},
|
|
79
|
+
quality: {
|
|
80
|
+
completeness: 0.98,
|
|
81
|
+
accuracy: 0.95,
|
|
82
|
+
timeliness: 0.99,
|
|
83
|
+
consistency: 0.97
|
|
84
|
+
},
|
|
85
|
+
owner: 'team-risk-analytics'
|
|
86
|
+
},
|
|
87
|
+
{
|
|
88
|
+
name: 'Customer 360 View',
|
|
89
|
+
description: 'Unified customer view combining KG relationships, transaction history, and demographic enrichment',
|
|
90
|
+
sources: ['kgdb', 'snowflake', 'bigquery'],
|
|
91
|
+
outputPort: '/api/v1/products/customer-360/query',
|
|
92
|
+
schema: {
|
|
93
|
+
columns: [
|
|
94
|
+
{ name: 'customer_id', type: 'STRING' },
|
|
95
|
+
{ name: 'full_name', type: 'STRING' },
|
|
96
|
+
{ name: 'relationships', type: 'ARRAY<STRING>' },
|
|
97
|
+
{ name: 'total_transactions', type: 'INT64' },
|
|
98
|
+
{ name: 'lifetime_value', type: 'DECIMAL(15,2)' }
|
|
99
|
+
]
|
|
100
|
+
},
|
|
101
|
+
quality: {
|
|
102
|
+
completeness: 0.96,
|
|
103
|
+
accuracy: 0.94
|
|
104
|
+
},
|
|
105
|
+
owner: 'team-customer-success'
|
|
106
|
+
},
|
|
107
|
+
{
|
|
108
|
+
name: 'Fraud Network Detection',
|
|
109
|
+
description: 'Graph-based fraud pattern detection using motif search and PageRank analysis',
|
|
110
|
+
sources: ['kgdb'],
|
|
111
|
+
outputPort: '/api/v1/products/fraud-networks/query',
|
|
112
|
+
schema: {
|
|
113
|
+
columns: [
|
|
114
|
+
{ name: 'network_id', type: 'STRING' },
|
|
115
|
+
{ name: 'entities', type: 'ARRAY<STRING>' },
|
|
116
|
+
{ name: 'pattern_type', type: 'STRING' },
|
|
117
|
+
{ name: 'risk_score', type: 'FLOAT64' },
|
|
118
|
+
{ name: 'pagerank', type: 'FLOAT64' }
|
|
119
|
+
]
|
|
120
|
+
},
|
|
121
|
+
quality: {
|
|
122
|
+
completeness: 0.92,
|
|
123
|
+
accuracy: 0.88
|
|
124
|
+
},
|
|
125
|
+
owner: 'team-fraud-detection'
|
|
126
|
+
}
|
|
127
|
+
]
|
|
128
|
+
|
|
129
|
+
for (const product of dataProducts) {
|
|
130
|
+
console.log(` Registering: ${product.name}`)
|
|
131
|
+
console.log(` Sources: ${product.sources.join(', ')}`)
|
|
132
|
+
console.log(` Owner: ${product.owner}`)
|
|
133
|
+
console.log(` Columns: ${product.schema.columns.length}`)
|
|
134
|
+
console.log(` Quality: completeness=${product.quality.completeness}, accuracy=${product.quality.accuracy}`)
|
|
135
|
+
console.log()
|
|
136
|
+
|
|
137
|
+
try {
|
|
138
|
+
const result = await federation.registerDataProduct(product)
|
|
139
|
+
console.log(` Registered with ID: ${result.id}`)
|
|
140
|
+
console.log()
|
|
141
|
+
} catch (error) {
|
|
142
|
+
console.log(` [Demo mode - Would register to catalog]`)
|
|
143
|
+
console.log()
|
|
144
|
+
}
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
// ============================================================================
|
|
148
|
+
// 2. List Catalog
|
|
149
|
+
// ============================================================================
|
|
150
|
+
console.log('2. List Data Products in Catalog')
|
|
151
|
+
console.log('═'.repeat(70))
|
|
152
|
+
|
|
153
|
+
try {
|
|
154
|
+
const catalog = await federation.listCatalog()
|
|
155
|
+
console.log(` Found ${catalog.length} data products:`)
|
|
156
|
+
for (const product of catalog) {
|
|
157
|
+
console.log(` - ${product.name} (${product.owner})`)
|
|
158
|
+
}
|
|
159
|
+
console.log()
|
|
160
|
+
} catch (error) {
|
|
161
|
+
console.log(` [Demo mode - Would list catalog entries]`)
|
|
162
|
+
console.log()
|
|
163
|
+
}
|
|
164
|
+
|
|
165
|
+
// ============================================================================
|
|
166
|
+
// 3. Filter Catalog
|
|
167
|
+
// ============================================================================
|
|
168
|
+
console.log('3. Filter Catalog by Owner')
|
|
169
|
+
console.log('═'.repeat(70))
|
|
170
|
+
|
|
171
|
+
try {
|
|
172
|
+
const riskProducts = await federation.listCatalog({ owner: 'team-risk-analytics' })
|
|
173
|
+
console.log(` Products owned by team-risk-analytics:`)
|
|
174
|
+
for (const product of riskProducts) {
|
|
175
|
+
console.log(` - ${product.name}`)
|
|
176
|
+
}
|
|
177
|
+
console.log()
|
|
178
|
+
} catch (error) {
|
|
179
|
+
console.log(` [Demo mode - Would filter by owner]`)
|
|
180
|
+
console.log()
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
// ============================================================================
|
|
184
|
+
// 4. DCAT DPROD Ontology Structure
|
|
185
|
+
// ============================================================================
|
|
186
|
+
console.log('4. DCAT DPROD Ontology Structure')
|
|
187
|
+
console.log('═'.repeat(70))
|
|
188
|
+
console.log(`
|
|
189
|
+
The catalog uses W3C DCAT-aligned DPROD ontology:
|
|
190
|
+
|
|
191
|
+
┌─────────────────────────────────────────────────────────────────────┐
|
|
192
|
+
│ dprod:DataProduct (High Risk Customer Analysis) │
|
|
193
|
+
├─────────────────────────────────────────────────────────────────────┤
|
|
194
|
+
│ dprod:name "High Risk Customer Analysis" │
|
|
195
|
+
│ dprod:description "Cross-domain risk scoring..." │
|
|
196
|
+
│ dprod:owner "team-risk-analytics" │
|
|
197
|
+
│ dprod:registeredIn <urn:catalog:hyperfederate> │
|
|
198
|
+
├─────────────────────────────────────────────────────────────────────┤
|
|
199
|
+
│ dprod:hasInputPort │
|
|
200
|
+
│ ├─ <urn:port:kgdb> (Knowledge Graph) │
|
|
201
|
+
│ ├─ <urn:port:snowflake> (Snowflake TPC-H) │
|
|
202
|
+
│ └─ <urn:port:bigquery> (BigQuery Public Data) │
|
|
203
|
+
│ │
|
|
204
|
+
│ dprod:hasOutputPort │
|
|
205
|
+
│ └─ <urn:port:api/products/high-risk-customers/query> │
|
|
206
|
+
├─────────────────────────────────────────────────────────────────────┤
|
|
207
|
+
│ dprod:hasSchema │
|
|
208
|
+
│ ├─ kg_entity: STRING │
|
|
209
|
+
│ ├─ risk_score: FLOAT64 │
|
|
210
|
+
│ ├─ entity_types: ARRAY<STRING> │
|
|
211
|
+
│ └─ ... │
|
|
212
|
+
├─────────────────────────────────────────────────────────────────────┤
|
|
213
|
+
│ dprod:hasQuality │
|
|
214
|
+
│ ├─ completeness: 0.98 │
|
|
215
|
+
│ ├─ accuracy: 0.95 │
|
|
216
|
+
│ ├─ timeliness: 0.99 │
|
|
217
|
+
│ └─ consistency: 0.97 │
|
|
218
|
+
├─────────────────────────────────────────────────────────────────────┤
|
|
219
|
+
│ dprod:hasLineage (W3C PROV) │
|
|
220
|
+
│ ├─ prov:wasGeneratedBy <urn:activity:federation-query> │
|
|
221
|
+
│ └─ prov:wasDerivedFrom <urn:source:kgdb>, <urn:source:snowflake> │
|
|
222
|
+
└─────────────────────────────────────────────────────────────────────┘
|
|
223
|
+
`)
|
|
224
|
+
|
|
225
|
+
// ============================================================================
|
|
226
|
+
// 5. Query Catalog via SPARQL
|
|
227
|
+
// ============================================================================
|
|
228
|
+
console.log('5. Query Catalog via SPARQL (Self-Describing)')
|
|
229
|
+
console.log('═'.repeat(70))
|
|
230
|
+
console.log(`
|
|
231
|
+
Since catalog is stored as RDF, we can query it via SPARQL:
|
|
232
|
+
|
|
233
|
+
SELECT ?product ?name ?owner ?quality WHERE {
|
|
234
|
+
?product a dprod:DataProduct ;
|
|
235
|
+
dprod:name ?name ;
|
|
236
|
+
dprod:owner ?owner ;
|
|
237
|
+
dprod:hasQuality ?q .
|
|
238
|
+
?q dprod:accuracy ?quality .
|
|
239
|
+
FILTER(?quality > 0.9)
|
|
240
|
+
}
|
|
241
|
+
|
|
242
|
+
Or via federated SQL:
|
|
243
|
+
|
|
244
|
+
SELECT * FROM graph_search('
|
|
245
|
+
PREFIX dprod: <https://gonnect.ai/domains/dprod#>
|
|
246
|
+
SELECT ?name ?owner ?accuracy WHERE {
|
|
247
|
+
?product a dprod:DataProduct ;
|
|
248
|
+
dprod:name ?name ;
|
|
249
|
+
dprod:owner ?owner ;
|
|
250
|
+
dprod:hasQuality [ dprod:accuracy ?accuracy ] .
|
|
251
|
+
FILTER(?accuracy > 0.9)
|
|
252
|
+
}
|
|
253
|
+
')
|
|
254
|
+
`)
|
|
255
|
+
|
|
256
|
+
// ============================================================================
|
|
257
|
+
// 6. ProofDAG with Catalog Evidence
|
|
258
|
+
// ============================================================================
|
|
259
|
+
console.log('6. ProofDAG with Catalog Evidence')
|
|
260
|
+
console.log('═'.repeat(70))
|
|
261
|
+
|
|
262
|
+
const proof = new ProofDAG('Data product catalog for enterprise risk analytics')
|
|
263
|
+
|
|
264
|
+
// Add catalog registration as evidence
|
|
265
|
+
const catalogNode = proof.addCatalogEvidence(
|
|
266
|
+
proof.rootId,
|
|
267
|
+
'High Risk Customer Analysis',
|
|
268
|
+
['kgdb', 'snowflake', 'bigquery'],
|
|
269
|
+
'dprod-001'
|
|
270
|
+
)
|
|
271
|
+
|
|
272
|
+
console.log(` Root claim: ${proof.rootClaim}`)
|
|
273
|
+
console.log(` Evidence type: catalog`)
|
|
274
|
+
console.log(` Proof hash: ${proof.computeHash()}`)
|
|
275
|
+
console.log()
|
|
276
|
+
|
|
277
|
+
// ============================================================================
|
|
278
|
+
// 7. Enterprise Governance Workflow
|
|
279
|
+
// ============================================================================
|
|
280
|
+
console.log('7. Enterprise Governance Workflow')
|
|
281
|
+
console.log('═'.repeat(70))
|
|
282
|
+
console.log(`
|
|
283
|
+
DPROD supports enterprise governance workflows:
|
|
284
|
+
|
|
285
|
+
┌──────────────────────────────────────────────────────────────┐
|
|
286
|
+
│ Data Product Lifecycle │
|
|
287
|
+
├──────────────────────────────────────────────────────────────┤
|
|
288
|
+
│ │
|
|
289
|
+
│ 1. DRAFT → Owner creates product definition │
|
|
290
|
+
│ │ │
|
|
291
|
+
│ ▼ │
|
|
292
|
+
│ 2. REVIEW → Data steward reviews schema + quality │
|
|
293
|
+
│ │ │
|
|
294
|
+
│ ▼ │
|
|
295
|
+
│ 3. APPROVED → Governance team approves for production │
|
|
296
|
+
│ │ │
|
|
297
|
+
│ ▼ │
|
|
298
|
+
│ 4. PUBLISHED → Product available in catalog │
|
|
299
|
+
│ │ │
|
|
300
|
+
│ ▼ │
|
|
301
|
+
│ 5. MONITORING → Quality metrics tracked continuously │
|
|
302
|
+
│ │
|
|
303
|
+
└──────────────────────────────────────────────────────────────┘
|
|
304
|
+
|
|
305
|
+
Approvals tracked via dprod:ApprovalWorkflow in KGDB.
|
|
306
|
+
`)
|
|
307
|
+
|
|
308
|
+
// ============================================================================
|
|
309
|
+
// Summary
|
|
310
|
+
// ============================================================================
|
|
311
|
+
console.log('╔══════════════════════════════════════════════════════════════════════╗')
|
|
312
|
+
console.log('║ Demo Complete! ║')
|
|
313
|
+
console.log('╚══════════════════════════════════════════════════════════════════════╝')
|
|
314
|
+
console.log(`
|
|
315
|
+
DCAT DPROD Catalog Benefits:
|
|
316
|
+
- Standards-Based: W3C DCAT alignment for interoperability
|
|
317
|
+
- Self-Describing: Stored as RDF, queryable via SPARQL
|
|
318
|
+
- Quality Tracking: Completeness, accuracy, timeliness metrics
|
|
319
|
+
- Lineage: W3C PROV-compatible provenance tracking
|
|
320
|
+
- Governance: Approval workflows for enterprise compliance
|
|
321
|
+
- Discovery: Filter by owner, sources, quality thresholds
|
|
322
|
+
|
|
323
|
+
Unique HyperFederate Capabilities:
|
|
324
|
+
- Catalog entries can include federated query definitions
|
|
325
|
+
- Virtual tables can be promoted to data products
|
|
326
|
+
- Full lineage across KGDB + Snowflake + BigQuery
|
|
327
|
+
- Real-time quality monitoring via SQL analytics
|
|
328
|
+
|
|
329
|
+
Architecture Principle:
|
|
330
|
+
- Catalog storage in Rust KGDB core (RDF native)
|
|
331
|
+
- SPARQL for self-describing queries
|
|
332
|
+
- TypeScript SDK provides thin RPC proxy layer
|
|
333
|
+
|
|
334
|
+
Audit Log Entries: ${federation.getAuditLog().length}
|
|
335
|
+
`)
|
|
336
|
+
}
|
|
337
|
+
|
|
338
|
+
// Run the demo
|
|
339
|
+
runDemo().catch(console.error)
|