@yottagraph-app/aether-instructions 1.1.8 → 1.1.10

package/skills/data-model/sanctions/DATA_DICTIONARY.md

@@ -0,0 +1,140 @@
+# Data Dictionary: Sanctions
+
+## Purpose
+
+Sanctioned persons and organizations from [OpenSanctions](https://www.opensanctions.org/), which aggregates sanctions lists from US OFAC, UK HM Treasury, EU, UN, Swiss SECO, and others. Input is OpenSanctions JSON (arrays or newline-delimited). Static reference data loaded in bulk. Only entities with `target: true` are ingested.
+
+`Record.Source`: `sanctions-source`
+
+---
+
+## Entity Types
+
+### `person`
+
+A sanctioned individual listed on one or more sanctions lists.
+
+- Primary key: OpenSanctions entity ID (`sanctions_id`)
+
+### `organization`
+
+A sanctioned company, legal entity, or other organization. OpenSanctions schema types `"Company"`, `"Organization"`, and `"LegalEntity"` all map here.
+
+- Primary key: OpenSanctions entity ID (`sanctions_id`)
+
+### `sanction_program`
+
+A government or international sanctions program. Created implicitly as relationship targets; resolved by strong ID (the program name).
+
+- Primary key: program name (e.g. `"OFAC-SDN"`, `"SECO-UKRAINE"`)
+
+### `country`
+
+Referenced by the `country` relationship on organization entities. Not created by this source.
+
+- Primary key: ISO country code
+
+---
+
+## Properties
+
+### Shared Properties (person + organization)
+
+#### Identity
+
+* `sanctions_id`
+  * Definition: unique identifier from OpenSanctions.
+  * Examples: `"ch-seco-94982"`, `"us-ofac-12345"`
+  * Derivation: `id` field from the JSON entity record.
+
+* `name`
+  * Definition: a known name for this entity. One atom per entry in the source `name` array.
+  * Examples: `"Zhongheng Lin"`, `"Acme Industries LLC"`
+  * Derivation: each entry in `properties.name`.
+
+* `alias`
+  * Definition: an alternative name or spelling. One atom per source alias.
+  * Examples: `"林仲恒 Lin"`, `"V. Putin"`
+  * Derivation: each entry in `properties.alias`.
+
+#### Sanctions Classification
+
+* `sanction_program`
+  * Definition: link to the sanctions program(s) under which this entity is designated. One atom per program.
+  * Examples: `"SECO-UKRAINE"`, `"OFAC-SDN"`, `"GB-RUS"`
+  * Derivation: each entry in `properties.programId`. Target `sanction_program` entity resolved by strong ID.
+
+* `sanctions_topic`
+  * Definition: classification tag for the type of designation. One atom per topic.
+  * Examples: `"sanction"`, `"poi"`, `"debarment"`, `"entity.associate"`
+  * Derivation: each entry in `properties.topics`.
+
+#### Context
+
+* `notes`
+  * Definition: context about why this entity is sanctioned. Multiple source notes concatenated with ` | `.
+  * Derivation: all entries from `properties.notes` concatenated.
+
+* `sanctioned`
+  * Definition: boolean flag (`1.0`) indicating a sanctions target. Emitted only when `notes` is absent.
+  * Derivation: fallback marker set to `1.0` when the entity has no notes.
+
+---
+
+### Person-Only Properties
+
+* `birth_date`
+  * Definition: date of birth. Only the first value is stored.
+  * Examples: `"1958-01-06"`, `"1952-10-07"`
+  * Derivation: `properties.birthDate[0]`. ISO 8601 date string.
+
+* `nationality`
+  * Definition: nationality as a two-letter ISO country code. Falls back to `country` if nationality is empty.
+  * Examples: `"cn"`, `"ru"`, `"us"`
+  * Derivation: `properties.nationality[0]`, falling back to `properties.country[0]`.
+
+* `position`
+  * Definition: job title, political role, or official position. Only the first value is stored.
+  * Examples: `"President of Russia"`, `"Owner of Shenzhen Biguang Technology"`
+  * Derivation: `properties.position[0]`.
+
+---
+
+### Organization-Only Properties
+
+* `country`
+  * Definition: link to the associated country entity. Only the first value is stored.
+  * Examples: `"ru"`, `"cn"`, `"ir"`
+  * Derivation: `properties.country[0]`. Target entity uses the ISO country code as name.
+
+* `address`
+  * Definition: physical address. Only the first value is stored.
+  * Examples: `"123 Main Street, Moscow, Russia"`, `"PO Box 456, Tehran"`
+  * Derivation: `properties.address[0]`.
+
+* `sector`
+  * Definition: business sector or industry classification. Only the first value is stored.
+  * Examples: `"Energy"`, `"Banking"`, `"Defense"`
+  * Derivation: `properties.sector[0]`.
+
+---
+
+### Sanction Program Properties
+
+None. Program entities carry only a strong ID (the program name) for resolution.
+
+---
+
+## Entity Relationships
+
+```
+person       ──[sanction_program]──→ sanction_program
+organization ──[sanction_program]──→ sanction_program
+organization ──[country]───────────→ country
+```
+
+---
+
+## Attributes
+
+None beyond standard atom metadata. Entity resolution snippets are carried on `Subject.NamedEntityResolverInformation.Snippets`.

package/skills/data-model/sanctions/schema.yaml

@@ -0,0 +1,153 @@
+# Dataset schema for sanctions data (OpenSanctions).
+#
+# This schema describes entity types, properties, and relationships produced
+# by atomizing structured JSON from the OpenSanctions aggregated dataset.
+# All elements are passive — they are created by the atomizer from parsed
+# JSON fields, not by LLM extraction.
+name: "sanctions"
+description: "Sanctioned persons, organizations, and their associated sanctions programs from the OpenSanctions aggregated dataset"
+
+extraction:
+  flavors: closed
+  properties: closed
+  relationships: closed
+  attributes: closed
+  events: closed
+
+flavors:
+  - name: "person"
+    description: "A real person as opposed to a fictional character, such as a CEO, politician, or public figure"
+    display_name: "Person"
+    mergeability: not_mergeable
+    passive: true
+
+  - name: "organization"
+    description: "A particular business, institution, or organization such as a corporation, university, government agency, or non-profit"
+    display_name: "Organization"
+    mergeability: not_mergeable
+    passive: true
+
+  - name: "sanction_program"
+    description: "A government or international sanctions program under which entities are designated"
+    display_name: "Sanction Program"
+    mergeability: not_mergeable
+    passive: true
+
+  - name: "country"
+    description: "A country referenced by sanctioned organizations, identified by ISO country code"
+    display_name: "Country"
+    mergeability: not_mergeable
+    passive: true
+
+properties:
+  - name: "sanctions_id"
+    type: string
+    description: "Unique identifier assigned to this entity by the upstream sanctions data provider (OpenSanctions entity ID)"
+    display_name: "Sanctions ID"
+    mergeability: not_mergeable
+    domain_flavors: ["person", "organization"]
+    examples: ["ch-seco-94982", "us-ofac-12345"]
+    passive: true
+
+  - name: "name"
+    type: string
+    description: "Display name of the entity"
+    display_name: "Name"
+    mergeability: not_mergeable
+    domain_flavors: ["person", "organization"]
+    examples: ["Zhongheng Lin", "Vladimir Putin"]
+    passive: true
+
+  - name: "alias"
+    type: string
+    description: "An alternative name or spelling for this entity, one atom per alias"
+    display_name: "Alias"
+    mergeability: not_mergeable
+    domain_flavors: ["person", "organization"]
+    passive: true
+
+  - name: "birth_date"
+    type: string
+    description: "Date of birth as reported in the sanctions listing, in ISO 8601 format"
+    display_name: "Birth Date"
+    mergeability: not_mergeable
+    domain_flavors: ["person"]
+    examples: ["1958-01-06", "1952-10-07"]
+    passive: true
+
+  - name: "nationality"
+    type: string
+    description: "Nationality or country affiliation as a two-letter ISO country code, falling back to the country field when nationality is not specified"
+    display_name: "Nationality"
+    mergeability: not_mergeable
+    domain_flavors: ["person"]
+    examples: ["cn", "ru", "us"]
+    passive: true
+
+  - name: "position"
+    type: string
+    description: "Job title, political role, or official position as stated in the sanctions listing"
+    display_name: "Position"
+    mergeability: not_mergeable
+    domain_flavors: ["person"]
+    examples: ["President of Russia", "Owner of Shenzhen Biguang Technology"]
+    passive: true
+
+  - name: "sanctions_topic"
+    type: string
+    description: "Classification tag indicating the type of sanctions designation, one atom per topic"
+    display_name: "Sanctions Topic"
+    mergeability: not_mergeable
+    domain_flavors: ["person", "organization"]
+    examples: ["sanction", "poi", "debarment", "entity.associate"]
+    passive: true
+
+  - name: "notes"
+    type: string
+    description: "Descriptive notes or commentary associated with the entity"
+    display_name: "Notes"
+    mergeability: not_mergeable
+    domain_flavors: ["person", "organization"]
+    passive: true
+
+  - name: "sanctioned"
+    type: float
+    description: "Boolean flag indicating this entity is a sanctions target, emitted only when no notes are present as a fallback marker"
+    display_name: "Sanctioned"
+    mergeability: not_mergeable
+    domain_flavors: ["person", "organization"]
+    passive: true
+
+  - name: "address"
+    type: string
+    description: "Physical address of the sanctioned organization as stated in the sanctions listing"
+    display_name: "Address"
+    mergeability: not_mergeable
+    domain_flavors: ["organization"]
+    passive: true
+
+  - name: "sector"
+    type: string
+    description: "Business sector or industry classification of the sanctioned organization"
+    display_name: "Sector"
+    mergeability: not_mergeable
+    domain_flavors: ["organization"]
+    examples: ["Energy", "Banking", "Defense"]
+    passive: true
+
+relationships:
+  - name: "sanction_program"
+    description: "Links a sanctioned entity to the sanctions program under which it is designated"
+    display_name: "Sanction Program"
+    mergeability: not_mergeable
+    domain_flavors: ["person", "organization"]
+    target_flavors: ["sanction_program"]
+    passive: true
+
+  - name: "country"
+    description: "Links a sanctioned organization to its associated country"
+    display_name: "Country"
+    mergeability: not_mergeable
+    domain_flavors: ["organization"]
+    target_flavors: ["country"]
+    passive: true

package/skills/data-model/stocks/DATA_DICTIONARY.md

@@ -0,0 +1,122 @@
+# Data Dictionary: Alpha Vantage Stocks
+
+## Source Overview
+
+Alpha Vantage stock market data via the `TIME_SERIES_DAILY` REST API. Provides daily OHLCV (open, high, low, close, volume) candlestick data for US-listed equities on NYSE, NASDAQ, and AMEX exchanges.
+
+Historical data available via `outputsize=full` (20+ years). Recent data (last 100 trading days) available via `outputsize=compact`. The streamer uses `full` for initial backfill and `compact` for daily incremental updates.
+
+Symbol universe: ~6,900 tickers (after filtering preferred stock symbols) sourced from the NASDAQ screener (static CSV, downloaded 2026-03-13). Symbols containing `^` (preferred shares) are excluded because Alpha Vantage does not support them.
+
+Rate limit: 1,200 requests/minute (premium plan). Downloads run concurrently (10 workers) with the Alpha Vantage client rate limiter gating throughput.
+
+| Pipeline | `Record.Source` |
+|----------|----------------|
+| Daily OHLCV | `stocks` |
+
+---
+
+## Entity Types
+
+### `financial_instrument`
+
+A publicly traded equity security (common stock, preferred stock, ETF, etc.) listed on a US exchange.
+
+- Primary key: `ticker_symbol` (e.g., `"AAPL"`, `"IBM"`, `"BRK-B"`)
+- Entity resolver: named entity, mergeable. Strong ID = `ticker_symbol`.
+- Entity name: ticker symbol (e.g., `"AAPL"`).
+
+### `organization`
+
+A company or entity whose equity securities are publicly traded on a US exchange.
+
+- Primary key: none (resolved by name).
+- Entity resolver: named entity, mergeable.
+- Entity name: full security name from the NASDAQ screener CSV (e.g., `"Apple Inc. Common Stock"`).
+- Note: for dual-class shares (e.g., GOOGL and GOOG), separate organization entities are created per share class. Entity resolution may merge these with organizations from other sources (e.g., EDGAR).
+
+---
+
+## Properties
+
+### Financial Instrument Properties
+
+#### Identity
+
+* `ticker_symbol`
+  * Definition: Stock ticker symbol as listed on the exchange.
+  * Examples: `"AAPL"`, `"IBM"`, `"MSFT"`, `"BRK-B"`
+  * Derivation: `symbol` field from the NASDAQ screener CSV; also present in Alpha Vantage response `Meta Data` → `2. Symbol`.
+
+* `company_name`
+  * Definition: Full registered security name.
+  * Examples: `"Apple Inc. Common Stock"`, `"International Business Machines Corporation Common Stock"`
+  * Derivation: `name` field from the NASDAQ screener CSV.
+
+* `exchange`
+  * Definition: US stock exchange where the security is listed.
+  * Examples: `"NYSE"`, `"NASDAQ"`, `"AMEX"`
+  * Derivation: `exchange` field from the NASDAQ screener CSV.
+
+* `sector`
+  * Definition: Market sector classification.
+  * Examples: `"Technology"`, `"Health Care"`, `"Industrials"`
+  * Derivation: `sector` field from the NASDAQ screener CSV. May be empty for some securities (e.g., SPACs, warrants).
+
+* `industry`
+  * Definition: Industry sub-classification within the sector.
+  * Examples: `"Computer Manufacturing"`, `"Biotechnology: Laboratory Analytical Instruments"`
+  * Derivation: `industry` field from the NASDAQ screener CSV. May be empty.
+
+#### Price Data (per daily candle)
+
+Each record represents one daily OHLCV candlestick for a single security. The atom timestamp is the trading day's date (Eastern time, converted to UTC).
+
+* `open_price`
+  * Definition: Opening price of the trading day. Unit: USD.
+  * Examples: `233.55`, `247.10`
+  * Derivation: `1. open` field from Alpha Vantage `Time Series (Daily)` response, parsed as float.
+
+* `high_price`
+  * Definition: Highest price reached during the trading day. Unit: USD.
+  * Examples: `234.00`, `250.05`
+  * Derivation: `2. high` field from Alpha Vantage response, parsed as float.
+
+* `low_price`
+  * Definition: Lowest price reached during the trading day. Unit: USD.
+  * Examples: `233.00`, `245.64`
+  * Derivation: `3. low` field from Alpha Vantage response, parsed as float.
+
+* `close_price`
+  * Definition: Closing price of the trading day. Unit: USD.
+  * Examples: `233.50`, `247.68`
+  * Derivation: `4. close` field from Alpha Vantage response, parsed as float.
+
+* `trading_volume`
+  * Definition: Number of shares traded during the day. Unit: shares.
+  * Examples: `1858`, `5547724`
+  * Derivation: `5. volume` field from Alpha Vantage response, parsed as integer (stored as float per v2 convention).
+
+---
+
+## Entity Relationships
+
+```
+organization ──[traded_as]──→ financial_instrument
+```
+
+* `traded_as`
+  * Definition: Links a company to the stock ticker under which its equity is publicly traded.
+  * Examples: `"Apple Inc. Common Stock" traded_as AAPL`, `"NVIDIA Corporation" traded_as NVDA`
+  * Derivation: one record per symbol, emitted alongside the price candle records. The organization name comes from the NASDAQ screener CSV `name` field; the financial_instrument is identified by the `ticker_symbol` strong ID.
+  * Note: one organization record with a `traded_as` relationship is emitted per download file (not per candle).
+
+---
+
+## Notes
+
+- **Timestamps**: Alpha Vantage returns timestamps in US/Eastern time. All atom timestamps are converted to UTC before storage.
+- **Adjusted prices**: The daily endpoint returns raw (as-traded) prices. Split/dividend-adjusted data is available via `TIME_SERIES_DAILY_ADJUSTED` (premium) but is not currently used.
+- **Missing data**: Not all symbols have data for all dates. Candles only exist for days with trading activity.
+- **Streaming model**: HandleStream runs as a continuous polling loop. First run downloads full history (`outputsize=full`). Subsequent runs download compact data (`outputsize=compact`, last 100 trading days) and atomize only candles newer than the checkpoint. Default poll interval: 24 hours.
+- **Preferred stock symbols**: Tickers containing `^` (e.g., `AHT^F`, `F^C`) are filtered out during symbol loading because Alpha Vantage returns empty responses for them.

package/skills/data-model/stocks/schema.yaml

@@ -0,0 +1,131 @@
+# Dataset schema for Alpha Vantage stock market data.
+#
+# This schema describes entity types and properties produced by atomizing
+# OHLCV (open, high, low, close, volume) candlestick data from the Alpha
+# Vantage TIME_SERIES_INTRADAY and TIME_SERIES_DAILY endpoints.
+#
+# All elements are passive -- created by the atomizer from structured JSON
+# fields, not by LLM extraction.
+name: "stocks"
+description: "US equity OHLCV price data from Alpha Vantage, covering NYSE, NASDAQ, and AMEX listed securities at 15-minute and daily intervals"
+
+extraction:
+  flavors: closed
+  properties: closed
+  relationships: closed
+  attributes: closed
+  events: closed
+
+flavors:
+  - name: "financial_instrument"
+    description: "A publicly traded equity security (common stock, preferred stock, ETF, etc.) listed on a US exchange, identified by its ticker symbol"
+    display_name: "Financial Instrument"
+    mergeability: not_mergeable
+    strong_id_properties: ["ticker_symbol"]
+    passive: true
+
+  - name: "organization"
+    description: "A company or entity whose equity securities are publicly traded on a US exchange"
+    display_name: "Organization"
+    mergeability: not_mergeable
+    passive: true
+
+properties:
+  # --- Identity ---
+
+  - name: "ticker_symbol"
+    type: string
+    description: "Stock ticker symbol as listed on the exchange (e.g., AAPL, IBM, BRK-B), from the NASDAQ screener and Alpha Vantage Meta Data"
+    display_name: "Ticker Symbol"
+    mergeability: not_mergeable
+    domain_flavors: ["financial_instrument"]
+    examples: ["AAPL", "IBM", "MSFT", "BRK-B"]
+    passive: true
+
+  - name: "company_name"
+    type: string
+    description: "Full registered security name from the NASDAQ screener"
+    display_name: "Company Name"
+    mergeability: not_mergeable
+    domain_flavors: ["financial_instrument"]
+    examples: ["Apple Inc. Common Stock", "International Business Machines Corporation Common Stock"]
+    passive: true
+
+  - name: "exchange"
+    type: string
+    description: "US stock exchange where the security is listed"
+    display_name: "Exchange"
+    mergeability: not_mergeable
+    domain_flavors: ["financial_instrument"]
+    examples: ["NYSE", "NASDAQ", "AMEX"]
+    passive: true
+
+  - name: "sector"
+    type: string
+    description: "Market sector classification from the NASDAQ screener; may be empty for SPACs, warrants, and other non-standard securities"
+    display_name: "Sector"
+    mergeability: not_mergeable
+    domain_flavors: ["financial_instrument"]
+    examples: ["Technology", "Health Care", "Industrials"]
+    passive: true
+
+  - name: "industry"
+    type: string
+    description: "Industry sub-classification within the sector from the NASDAQ screener; may be empty"
+    display_name: "Industry"
+    mergeability: not_mergeable
+    domain_flavors: ["financial_instrument"]
+    examples: ["Computer Manufacturing", "Biotechnology: Laboratory Analytical Instruments"]
+    passive: true
+
+  # --- Price Data (per candle) ---
+
+  - name: "open_price"
+    type: float
+    description: "Opening price of the trading interval in USD, from Alpha Vantage field '1. open'"
+    display_name: "Open Price"
+    mergeability: not_mergeable
+    domain_flavors: ["financial_instrument"]
+    passive: true
+
+  - name: "high_price"
+    type: float
+    description: "Highest price reached during the trading interval in USD, from Alpha Vantage field '2. high'"
+    display_name: "High Price"
+    mergeability: not_mergeable
+    domain_flavors: ["financial_instrument"]
+    passive: true
+
+  - name: "low_price"
+    type: float
+    description: "Lowest price reached during the trading interval in USD, from Alpha Vantage field '3. low'"
+    display_name: "Low Price"
+    mergeability: not_mergeable
+    domain_flavors: ["financial_instrument"]
+    passive: true
+
+  - name: "close_price"
+    type: float
+    description: "Closing price of the trading interval in USD, from Alpha Vantage field '4. close'"
+    display_name: "Close Price"
+    mergeability: not_mergeable
+    domain_flavors: ["financial_instrument"]
+    passive: true
+
+  - name: "trading_volume"
+    type: float
+    description: "Number of shares traded during the interval, from Alpha Vantage field '5. volume'; stored as float per v2 convention"
+    display_name: "Trading Volume"
+    mergeability: not_mergeable
+    domain_flavors: ["financial_instrument"]
+    passive: true
+
+relationships:
+  - name: "traded_as"
+    description: "Links a company (organization) to the financial instrument (stock ticker) under which its equity is publicly traded"
+    display_name: "Traded As"
+    mergeability: not_mergeable
+    domain_flavors: ["organization"]
+    target_flavors: ["financial_instrument"]
+    examples: ["Apple Inc. traded as AAPL", "International Business Machines Corporation traded as IBM"]
+    passive: true

package/skills/data-model/system_schema.yaml

@@ -0,0 +1,71 @@
+# System schema: universal infrastructure elements for all datasets.
+#
+# These elements are created by the atomizer, not by LLM extraction. Every
+# dataset gets them for free via WithSystemSchema(). The schema:: prefix
+# follows the meta-schema naming convention from the schemaless design
+# (experimental/jon/schemaless/DESIGN.md) and prevents collision with
+# dataset-defined elements.
+#
+# This schema should rarely change. Dataset-specific elements (source
+# flavors, publications, topics, trustworthiness, etc.) belong in the
+# dataset schema.
+
+name: "system"
+description: "Universal infrastructure elements for all datasets"
+
+flavors:
+  - name: "schema::flavor::event"
+    description: "A significant occurrence extracted from source text"
+    display_name: "Event"
+    mergeability: not_mergeable
+
+relationships:
+  - name: "schema::relationship::participant"
+    description: "An entity participates in an event"
+    display_name: "Participant"
+    mergeability: not_mergeable
+    domain_flavors: ["schema::flavor::event"]
+
+properties:
+  - name: "schema::property::event_category"
+    type: string
+    description: "The category of an event (e.g., Bankruptcy, IPO, Layoffs)"
+    display_name: "Event Category"
+    mergeability: not_mergeable
+    domain_flavors: ["schema::flavor::event"]
+
+  - name: "schema::property::event_likelihood"
+    type: string
+    description: "Temporal status of an event: confirmed, ongoing, likely, or speculative"
+    display_name: "Event Likelihood"
+    mergeability: not_mergeable
+    domain_flavors: ["schema::flavor::event"]
+
+  - name: "schema::property::event_date"
+    type: string
+    description: "The date an event occurred in YYYY-MM-DD format (or YYYY-MM or YYYY for partial dates)"
+    display_name: "Event Date"
+    mergeability: not_mergeable
+    domain_flavors: ["schema::flavor::event"]
+
+  - name: "schema::property::event_description"
+    type: string
+    description: "A detailed, objective description of an event"
+    display_name: "Event Description"
+    mergeability: not_mergeable
+    domain_flavors: ["schema::flavor::event"]
+
+attributes:
+  - property: "schema::relationship::participant"
+    name: "role"
+    type: string
+    description: "The entity's role in the event (e.g., acquirer, target, plaintiff)"
+    display_name: "Role"
+    mergeability: not_mergeable
+
+  - property: "schema::relationship::participant"
+    name: "sentiment"
+    type: float
+    description: "Impact score for this entity's involvement in the event"
+    display_name: "Sentiment"
+    mergeability: not_mergeable