@secondlayer/subgraphs 3.2.1 → 3.3.0

package/README.md

@@ -93,6 +93,87 @@ Delivery bodies and response previews land in `subscription_deliveries`. Rows wh
 | `SUBGRAPH_REINDEX_BATCH_SIZE` | plan-based | Override the default historical block batch size used by reindex/backfill. |
 | `SUBGRAPH_REINDEX_MIN_BATCH_SIZE` | plan-based | Override the adaptive lower bound for reindex/backfill batches. |
 | `SUBGRAPH_REINDEX_MAX_BATCH_SIZE` | plan-based | Override the adaptive upper bound for reindex/backfill batches. |
+| `DATABASE_MAX_POOLS` | `25` | Max cached connection pools. With BYO subgraphs (one pool per user DB) the least-recently-used pool is evicted past this cap; the source/target pools are never evicted. |
+| `DATABASE_IDLE_TIMEOUT` | `300` | Seconds before idle connections are closed (`0` = never). Keeps a fleet of BYO pools from pinning connections. |
+
+## Bring your own database (BYO data plane)
+
+There are three tiers, not two:
+
+| Tier | Indexer / decode | Handler exec | Database | Serving API |
+| --- | --- | --- | --- | --- |
+| Managed (default) | ours | ours | ours | ours |
+| **BYO data plane** | ours | ours | **yours** | **yours** |
+| Self-host | yours | yours | yours | yours |
+
+With BYO, the managed pipeline (ingest → decode → match → run your handler) is
+unchanged, but your handler's rows land in **your** Postgres and the serving API
+reads from there. Deploy with a connection string:
+
+```bash
+sl subgraphs deploy subgraphs/my.ts --database-url "postgres://user:pass@your-host:5432/db"
+```
+
+The connection string is stored encrypted at rest (AES-GCM, keyed by
+`SECONDLAYER_SECRETS_KEY`) and never returned in API responses. The server
+verifies the connection before deploying. Once deployed, query the
+`subgraph_…` schema directly with any ORM/GraphQL/REST — we're no longer in the
+serving path.
+
+**Preview before it touches your DB.** `POST /api/subgraphs` with
+`{"dryRun": true, "databaseUrl": "…"}` returns the exact DDL plus a grant
+script and verifies the connection — without writing anything.
+
+**Constraints (v1):**
+
+- **Idempotent handlers only.** A BYO block write can't share the managed
+  transaction, so a crash replays the block (at-least-once). `ctx.insert` and
+  `ctx.upsert` (with a unique key) are safe — flush is replace-per-height. A
+  deploy with non-idempotent `ctx.update` / `ctx.patchOrInsert` is rejected.
+- **No reindex.** Reindex would drop + rebuild the schema in your DB from a
+  background job; instead, re-deploy to rebuild (or drop the schema yourself).
+- **Delete leaves your data.** Deleting the subgraph removes our registry row
+  (and the stored connection) and pauses subscriptions, but never drops the
+  schema in your database.
+
+**Recommended:** give Secondlayer a least-privilege role scoped to its own
+schema, and point `--database-url` at a session-mode pooler endpoint
+(PgBouncer/Neon/Supabase) rather than raw Postgres.
+
+### ORM codegen
+
+Once rows land in your DB, generate a typed schema for your ORM:
+
+```bash
+sl subgraphs codegen subgraphs/my.ts --target prisma  -o prisma/schema.prisma
+sl subgraphs codegen subgraphs/my.ts --target drizzle -o db/schema.ts
+```
+
+Prisma and Drizzle have first-class generators (`generatePrismaSchema` /
+`generateDrizzleSchema` are exported); both emit relations/`@relation` from the
+schema's `relations` metadata. For Kysely, run `kysely-codegen` against the DB.
+Output mirrors the deployed DDL — `prisma db pull` should be a no-op; treat the
+tables as read-only (the processor owns them) and never `migrate`/`push`.
+`uint`→`Decimal`/`numeric` and the `BigInt` id need `.toString()` for JSON.
+
+## Trait-scoped sources
+
+A source can target a SIP standard instead of a fixed contract — it indexes
+every contract the registry classifies as that standard (incl. ones deployed
+later):
+
+```ts
+sources: {
+  tokens: { type: "ft_transfer", trait: "sip-010" }, // all SIP-010 tokens
+}
+```
+
+`trait` (`sip-009` | `sip-010` | `sip-013`) is supported on FT/NFT/`contract_call`/
+`print_event` filters and composes (AND) with other fields. Token filters match
+the asset-identifier's contract; `contract_call`/`print` match `contract_id`.
+Resolution is as-of-block, so a reindex backfills a contract's full history even
+if it was classified after deploy. Requires the contract registry to be
+populated. Discover the set via `GET /v1/contracts?trait=sip-010`.
 
 ## Postgres + pool mode
 

package/dist/src/index.d.ts

@@ -9,6 +9,22 @@ interface SubgraphColumn {
 	search?: boolean;
 	default?: string | number | boolean;
 }
+/**
+* A foreign-key relation to another table in the same subgraph. Drives DDL FK
+* constraints and ORM codegen (`@relation` in Prisma, `relations()` in Drizzle)
+* so generated clients get typed joins. The referenced columns must form a
+* `uniqueKeys` entry on the target table.
+*/
+interface SubgraphRelation {
+	/** Relation field name on this table's generated model (e.g. "pool"). */
+	name: string;
+	/** Target table name in this subgraph. */
+	references: string;
+	/** Local column(s) holding the foreign key. */
+	fields: string[];
+	/** Target column(s) the fields point at (a uniqueKeys entry on the target). */
+	referencedColumns: string[];
+}
 /** Table definition within a subgraph schema */
 interface SubgraphTable {
 	columns: Record<string, SubgraphColumn>;
@@ -16,6 +32,8 @@ interface SubgraphTable {
 	indexes?: string[][];
 	/** Unique key constraints (each entry is an array of column names). Required for upsert. */
 	uniqueKeys?: string[][];
+	/** Foreign-key relations to other tables (for typed ORM joins). */
+	relations?: SubgraphRelation[];
 }
 /** Subgraph schema — maps table names to table definitions */
 type SubgraphSchema = Record<string, SubgraphTable>;
@@ -42,39 +60,47 @@ interface StxLockFilter {
 	lockedAddress?: string;
 	minAmount?: bigint;
 }
+/**
+* Restrict a source to contracts conforming to a trait/standard (e.g. "sip-010")
+* instead of a fixed contract — resolved from the contract registry at match time,
+* as-of each processed block. Lets a source index "all SIP-010 tokens" etc.
+*/
+type TraitScope = {
+	trait?: string
+};
 /** FT event filters */
-interface FtTransferFilter {
+interface FtTransferFilter extends TraitScope {
 	type: "ft_transfer";
 	assetIdentifier?: string;
 	sender?: string;
 	recipient?: string;
 	minAmount?: bigint;
 }
-interface FtMintFilter {
+interface FtMintFilter extends TraitScope {
 	type: "ft_mint";
 	assetIdentifier?: string;
 	recipient?: string;
 	minAmount?: bigint;
 }
-interface FtBurnFilter {
+interface FtBurnFilter extends TraitScope {
 	type: "ft_burn";
 	assetIdentifier?: string;
 	sender?: string;
 	minAmount?: bigint;
 }
 /** NFT event filters */
-interface NftTransferFilter {
+interface NftTransferFilter extends TraitScope {
 	type: "nft_transfer";
 	assetIdentifier?: string;
 	sender?: string;
 	recipient?: string;
 }
-interface NftMintFilter {
+interface NftMintFilter extends TraitScope {
 	type: "nft_mint";
 	assetIdentifier?: string;
 	recipient?: string;
 }
-interface NftBurnFilter {
+interface NftBurnFilter extends TraitScope {
 	type: "nft_burn";
 	assetIdentifier?: string;
 	sender?: string;
@@ -121,7 +147,7 @@ interface ContractCallEvent {
 	};
 }
 /** Contract event filters */
-interface ContractCallFilter {
+interface ContractCallFilter extends TraitScope {
 	type: "contract_call";
 	contractId?: string;
 	functionName?: string;
@@ -139,7 +165,7 @@ interface ContractDeployFilter {
 	deployer?: string;
 	contractName?: string;
 }
-interface PrintEventFilter {
+interface PrintEventFilter extends TraitScope {
 	type: "print_event";
 	contractId?: string;
 	topic?: string;
@@ -559,6 +585,22 @@ interface GeneratedSQL {
 * each with auto-columns and indexes.
 */
 declare function generateSubgraphSQL(def: SubgraphDefinition, schemaNameOverride?: string): GeneratedSQL;
+interface PrismaGenOptions {
+	/** Postgres schema the tables live in (account-scoped). */
+	schemaName?: string;
+	/** env var the datasource url reads from. */
+	datasourceEnv?: string;
+	/**
+	* Emit only the `model` blocks (no datasource/generator). Lets users compose
+	* these models with their own schema via Prisma's `prismaSchemaFolder`.
+	*/
+	modelsOnly?: boolean;
+}
+declare function generatePrismaSchema(def: SubgraphDefinition, opts?: PrismaGenOptions): string;
+interface DrizzleGenOptions {
+	schemaName?: string;
+}
+declare function generateDrizzleSchema(def: SubgraphDefinition, opts?: DrizzleGenOptions): string;
 import { pgSchemaName } from "@secondlayer/shared/db/queries/subgraphs";
 import { Database } from "@secondlayer/shared/db";
 import { Kysely } from "kysely";
@@ -586,6 +628,18 @@ interface DeployDiff {
 	addedColumns: Record<string, string[]>;
 	breakingChanges: string[];
 }
+interface DeployPlan {
+	schemaName: string;
+	/** DDL Secondlayer will run against your database. */
+	statements: string[];
+	/** Least-privilege grant script to run once, before deploying. */
+	grantScript: string;
+}
+/**
+* Render the DDL + grant script a BYO deploy would run, without executing.
+* Powers `--dry-run`: the user reviews exactly what touches their DB first.
+*/
+declare function renderDeployPlan(def: SubgraphDefinition, schemaName?: string): DeployPlan;
 /**
 * Deploy a subgraph schema to the database.
 * - New subgraph → CREATE SCHEMA + tables + register
@@ -601,10 +655,18 @@ declare function deploySchema(db: AnyDb, def: SubgraphDefinition, handlerPath: s
 	version?: string
 	handlerCode?: string
 	sourceCode?: string
+	/**
+	* BYO data plane: when set, schema DDL (CREATE/ALTER/index) runs against
+	* the user-owned DB while the subgraphs registry row stays on `db`
+	* (managed). Defaults to `db` — managed deploys are unchanged.
+	*/
+	dataDb?: AnyDb
+	/** Encrypted user-DB connection string to persist on the registry row. */
+	databaseUrlEnc?: Buffer | null
 }): Promise<{
 	action: "created" | "unchanged" | "handler_updated" | "updated" | "reindexed"
 	subgraphId: string
 	version: string
 	diff?: DeployDiff
 }>;
-export { validateSubgraphDefinition, resumeReindex, reindexSubgraph, pgSchemaName, generateSubgraphSQL, diffSchema, deploySchema, defineSubgraph, backfillSubgraph, WriteRow, WhereInput, TypedSubgraphDefinition, TypedSubgraphContext, TypedHandlers, TxMeta, TableDiff, SystemRow, SubgraphTableClient, SubgraphTable, SubgraphSchema, SubgraphHandler, SubgraphFilter, SubgraphDefinition, SubgraphContext, SubgraphColumn, StxTransferPayload, StxTransferFilter, StxMintPayload, StxMintFilter, StxLockPayload, StxLockFilter, StxBurnPayload, StxBurnFilter, RowValue, ReindexOptions, PrintEventPayload, PrintEventFor, PrintEventFilter, NftTransferPayload, NftTransferFilter, NftMintPayload, NftMintFilter, NftBurnPayload, NftBurnFilter, InferTableRow, InferSubgraphClient, InferColumnType, GeneratedSQL, FtTransferPayload, FtTransferFilter, FtMintPayload, FtMintFilter, FtBurnPayload, FtBurnFilter, FindManyOptions, EventForFilter, ContractDeployPayload, ContractDeployFilter, ContractCallPayload, ContractCallFilter, ContractCallEvent, ComputedValue, ComparisonFilter, ColumnType, ColumnToTS, ColumnDiff, AnyEvent };
+export { validateSubgraphDefinition, resumeReindex, renderDeployPlan, reindexSubgraph, pgSchemaName, generateSubgraphSQL, generatePrismaSchema, generateDrizzleSchema, diffSchema, deploySchema, defineSubgraph, backfillSubgraph, WriteRow, WhereInput, TypedSubgraphDefinition, TypedSubgraphContext, TypedHandlers, TxMeta, TableDiff, SystemRow, SubgraphTableClient, SubgraphTable, SubgraphSchema, SubgraphHandler, SubgraphFilter, SubgraphDefinition, SubgraphContext, SubgraphColumn, StxTransferPayload, StxTransferFilter, StxMintPayload, StxMintFilter, StxLockPayload, StxLockFilter, StxBurnPayload, StxBurnFilter, RowValue, ReindexOptions, PrismaGenOptions, PrintEventPayload, PrintEventFor, PrintEventFilter, NftTransferPayload, NftTransferFilter, NftMintPayload, NftMintFilter, NftBurnPayload, NftBurnFilter, InferTableRow, InferSubgraphClient, InferColumnType, GeneratedSQL, FtTransferPayload, FtTransferFilter, FtMintPayload, FtMintFilter, FtBurnPayload, FtBurnFilter, FindManyOptions, EventForFilter, DrizzleGenOptions, DeployPlan, ContractDeployPayload, ContractDeployFilter, ContractCallPayload, ContractCallFilter, ContractCallEvent, ComputedValue, ComparisonFilter, ColumnType, ColumnToTS, ColumnDiff, AnyEvent };