@pikaa-ai/pikaa 0.2.5 → 0.3.0

package/skills/database-lookup/references/datacommons.md

@@ -0,0 +1,237 @@
+# Google Data Commons API
+
+## Base URL
+
+```
+https://api.datacommons.org
+```
+
+## Authentication
+
+**API key required.** Obtain from the Google Cloud Console (enable the Data Commons API).
+
+Pass as query parameter: `&key=YOUR_KEY`
+
+Or as header: `X-API-Key: YOUR_KEY`
+
+Note: Many endpoints work without a key for light usage, but a key is recommended for reliable access.
+
+## Key Endpoints
+
+### 1. Get Statistical Value (single observation)
+```
+GET /v2/observation
+```
+| Parameter    | Required | Description                                                |
+|-------------|----------|------------------------------------------------------------|
+| key          | Yes      | API key                                                    |
+| entity.dcids | Yes     | Place DCID(s) (e.g., `country/USA`, `geoId/06`)          |
+| variable.dcids| Yes    | Statistical variable DCID(s)                               |
+| date         | No       | Specific date or `LATEST`                                 |
+| select       | No       | Fields to select: `entity`, `variable`, `date`, `value`   |
+
+Example:
+```
+https://api.datacommons.org/v2/observation?key=YOUR_KEY&entity.dcids=country/USA&variable.dcids=Count_Person&date=LATEST&select=entity&select=variable&select=date&select=value
+```
+
+### 2. Get Statistical Time Series
+```
+GET /v2/observation
+```
+Use same endpoint but omit `date` parameter (or set `date=''`) to get the full time series.
+
+Example (population time series for USA):
+```
+https://api.datacommons.org/v2/observation?key=YOUR_KEY&entity.dcids=country/USA&variable.dcids=Count_Person&select=entity&select=variable&select=date&select=value
+```
+
+### 3. Node Info (property values of an entity)
+```
+GET /v2/node
+```
+| Parameter | Required | Description                                        |
+|-----------|----------|----------------------------------------------------|
+| key       | Yes      | API key                                            |
+| nodes     | Yes      | DCID(s) of the node                               |
+| property  | Yes      | Property expression: `->prop` (out), `<-prop` (in)|
+
+Example (get properties of California):
+```
+https://api.datacommons.org/v2/node?key=YOUR_KEY&nodes=geoId/06&property=->*
+```
+
+Example (get name of a place):
+```
+https://api.datacommons.org/v2/node?key=YOUR_KEY&nodes=geoId/06&property=->name
+```
+
+### 4. SPARQL Query
+```
+POST /v2/sparql
+```
+Content-Type: `application/json`
+
+Body:
+```json
+{
+  "query": "SELECT ?name WHERE { ?state typeOf State . ?state name ?name . ?state containedInPlace country/USA }"
+}
+```
+
+Pass API key as query param or header.
+
+Example (curl):
+```
+curl -X POST 'https://api.datacommons.org/v2/sparql?key=YOUR_KEY' \
+  -H 'Content-Type: application/json' \
+  -d '{"query": "SELECT ?name WHERE { ?place typeOf Country . ?place name ?name } LIMIT 10"}'
+```
+
+### 5. Resolve Entities (map names/coords to DCIDs)
+```
+GET /v2/resolve
+```
+| Parameter  | Required | Description                                    |
+|------------|----------|------------------------------------------------|
+| key        | Yes      | API key                                        |
+| nodes      | Yes      | Entity identifiers to resolve                 |
+| property   | Yes      | `<-description` (name lookup) or coordinate-based |
+
+Example (resolve by name):
+```
+https://api.datacommons.org/v2/resolve?key=YOUR_KEY&nodes=California&property=<-description->dcid
+```
+
+### 6. Search for Statistical Variables
+```
+GET /v2/variable/search
+```
+| Parameter | Required | Description            |
+|-----------|----------|------------------------|
+| key       | Yes      | API key                |
+| query     | Yes      | Search keywords        |
+
+Example:
+```
+https://api.datacommons.org/v2/variable/search?key=YOUR_KEY&query=unemployment+rate
+```
+
+## Common DCIDs
+
+### Places
+| DCID              | Description          |
+|-------------------|----------------------|
+| country/USA       | United States        |
+| country/GBR       | United Kingdom       |
+| country/CHN       | China                |
+| geoId/06          | California           |
+| geoId/0667000     | San Francisco city   |
+| geoId/06085       | Santa Clara County   |
+
+### Statistical Variables
+| DCID                                    | Description                    |
+|-----------------------------------------|--------------------------------|
+| Count_Person                            | Total population               |
+| Count_Person_Employed                   | Employed persons               |
+| UnemploymentRate_Person                 | Unemployment rate              |
+| Median_Income_Person                    | Median income                  |
+| Amount_EconomicActivity_GrossDomesticProduction_Nominal | Nominal GDP     |
+| Mean_ConsumerPriceIndex                 | Consumer price index           |
+| Count_Death                             | Number of deaths               |
+| Count_Person_BelowPovertyLevelInThePast12Months | Persons in poverty  |
+| Median_Age_Person                       | Median age                     |
+
+## Response Format
+
+### Observation response
+```json
+{
+  "byVariable": {
+    "Count_Person": {
+      "byEntity": {
+        "country/USA": {
+          "orderedFacets": [
+            {
+              "facetId": "2176550201",
+              "observations": [
+                {
+                  "date": "2020",
+                  "value": 331449281
+                },
+                {
+                  "date": "2021",
+                  "value": 331893745
+                }
+              ]
+            }
+          ]
+        }
+      }
+    }
+  },
+  "facets": {
+    "2176550201": {
+      "importName": "CensusACS5YearSurvey",
+      "provenanceUrl": "https://www.census.gov/",
+      "measurementMethod": "CensusACS5yrSurvey"
+    }
+  }
+}
+```
+
+### Node response
+```json
+{
+  "data": {
+    "geoId/06": {
+      "arcs": {
+        "name": {
+          "nodes": [
+            {
+              "value": "California"
+            }
+          ]
+        }
+      }
+    }
+  }
+}
+```
+
+### SPARQL response
+```json
+{
+  "header": ["?name"],
+  "rows": [
+    { "cells": [{ "value": "Alabama" }] },
+    { "cells": [{ "value": "Alaska" }] }
+  ]
+}
+```
+
+### Variable search response
+```json
+{
+  "variables": [
+    {
+      "dcid": "UnemploymentRate_Person",
+      "displayName": "Unemployment Rate"
+    }
+  ]
+}
+```
+
+## Rate Limits
+
+- Without API key: very limited (roughly a few requests per minute; may be blocked).
+- With API key: not formally published, but generally generous for normal use.
+- Implement client-side throttling (1-2 requests/second recommended).
+- Bulk data available via the Data Commons data download for large-scale analysis.
+
+## Notes
+
+- The V2 API (paths starting with `/v2/`) is the current recommended version.
+- Older V1 endpoints (`/v1/bulk/observations/series`, `/stat/value`, etc.) still work but are deprecated.
+- DCID = Data Commons Identifier. Every entity, statistical variable, and concept has a unique DCID.
+- The knowledge graph includes data from US Census, World Bank, CDC, BLS, FBI, and many other sources.

package/skills/database-lookup/references/dbsnp.md

@@ -0,0 +1,143 @@
+# dbSNP API Reference
+
+## Overview
+SNP and variant data. Accessible via two APIs: NCBI E-utilities (`db=snp`) for search/metadata, and the NCBI Variation Services REST API for detailed variant annotations.
+
+## Base URLs
+```
+E-utilities:  https://eutils.ncbi.nlm.nih.gov/entrez/eutils/
+Variation API: https://api.ncbi.nlm.nih.gov/variation/v0/
+```
+
+## Authentication
+- **E-utilities**: API key recommended (`&api_key=KEY`). 3 req/sec without, 10 req/sec with key.
+- **Variation API**: No auth required. Rate limits apply (undocumented; be respectful, ~1-2 req/sec).
+
+---
+
+## E-utilities Endpoints (db=snp)
+
+### 1. ESearch -- Search SNPs
+```
+GET esearch.fcgi?db=snp&term=QUERY&retmax=N&retmode=json
+```
+
+**Example -- search SNPs in BRCA1 gene:**
+```
+GET esearch.fcgi?db=snp&term=BRCA1[Gene Name] AND homo sapiens[Organism]&retmax=5&retmode=json
+```
+Response:
+```json
+{
+  "esearchresult": {
+    "count": "12847",
+    "idlist": ["80357713", "80357508", ...]
+  }
+}
+```
+Note: IDs returned are rs numbers without the "rs" prefix.
+
+### 2. ESummary -- SNP summaries
+```
+GET esummary.fcgi?db=snp&id=IDS&retmode=json
+```
+
+**Example -- get summary for rs334 (sickle cell variant):**
+```
+GET esummary.fcgi?db=snp&id=334&retmode=json
+```
+Response includes: `snp_id`, `chr`, `chrpos`, `genes`, `clinical_significance`, `global_mafs`, `docsum`.
+
+### 3. EFetch -- Fetch SNP details (XML only)
+```
+GET efetch.fcgi?db=snp&id=IDS&rettype=json&retmode=text
+```
+Note: EFetch for dbSNP returns JSON with `rettype=json`. Also supports XML with `retmode=xml`.
+
+---
+
+## Variation Services API
+
+### 1. Lookup variant by rsID
+```
+GET /variation/v0/refsnp/{rsid}
+```
+
+**Example:**
+```
+GET https://api.ncbi.nlm.nih.gov/variation/v0/refsnp/334
+```
+Response (JSON, abbreviated):
+```json
+{
+  "refsnp_id": "334",
+  "create_date": "2000/09/19",
+  "primary_snapshot_data": {
+    "placements_with_allele": [...],
+    "allele_annotations": [...],
+    "support": [...]
+  },
+  "present_obs_movements": [
+    {
+      "component_ids": [{"type": "clinvar", "value": "..."}],
+      "observation": {
+        "seq_id": "NC_000011.10",
+        "position": 5227002,
+        "deleted_sequence": "T",
+        "inserted_sequence": "A"
+      }
+    }
+  ]
+}
+```
+
+### 2. Lookup variant by SPDI notation
+```
+GET /variation/v0/spdi/{spdi}/rsids
+```
+SPDI format: `SeqID:Position:Deletion:Insertion`
+
+**Example:**
+```
+GET https://api.ncbi.nlm.nih.gov/variation/v0/spdi/NC_000011.10:5227002:T:A/rsids
+```
+
+### 3. Lookup variant by HGVS
+```
+GET /variation/v0/hgvs/{hgvs}/contextuals
+```
+
+**Example:**
+```
+GET https://api.ncbi.nlm.nih.gov/variation/v0/hgvs/NC_000011.10:g.5227003T>A/contextuals
+```
+
+### 4. Batch rsID lookup (POST)
+```
+POST /variation/v0/refsnp/batch
+Content-Type: application/json
+
+{"refsnp_ids": ["334", "1805007", "7412"]}
+```
+
+## Common E-utilities Search Patterns
+```
+# By rs number
+term=334[RS ID]
+
+# Clinical significance
+term=pathogenic[Clinical Significance] AND BRCA1[Gene Name]
+
+# By chromosome position (GRCh38)
+term=11[Chromosome] AND 5227002:5227002[Base Position]
+
+# By variant type
+term=missense[Function Class] AND TP53[Gene Name]
+
+# By global minor allele frequency
+term=0.01:0.05[Global MAF]
+```
+
+## Rate Limits
+- E-utilities: 3 req/sec (no key), 10 req/sec (with key)
+- Variation Services API: No published limit; recommend 1-2 req/sec

package/skills/database-lookup/references/disgenet.md

@@ -0,0 +1,52 @@
+# DisGeNET (Gene-Disease Associations)
+
+## Base URL
+```
+https://www.disgenet.org/api
+```
+
+## Auth
+**API key required.** Register at disgenet.org, then authenticate:
+```bash
+curl -X POST https://www.disgenet.org/api/auth/ \
+  -d 'email=you@example.com&password=yourpassword'
+# Returns: {"token": "abc123..."}
+```
+Pass as: `Authorization: Bearer <token>`
+
+Load token from `.env` as `DISGENET_API_KEY`.
+
+## Key Endpoints
+
+| Endpoint | Description |
+|----------|-------------|
+| `/gda/gene/{gene_id}` | Gene-disease associations (NCBI gene ID) |
+| `/gda/disease/{disease_id}` | Gene-disease associations (UMLS CUI) |
+| `/gda/evidences/gene/{gene_id}` | Evidence-level data |
+| `/vda/gene/{gene_id}` | Variant-disease associations for a gene |
+| `/vda/variant/{rsid}` | Variant-disease associations (dbSNP rsID) |
+
+## Parameters
+- `source` — `CURATED`, `BEFREE`, `ALL`
+- `min_score` — GDA score threshold (0-1)
+- `min_ei` — evidence index threshold
+- `format` — `json` or `tsv`
+- `limit`, `offset` — pagination
+
+## Example Calls
+```
+# Gene-disease for TP53 (gene ID 7157)
+/gda/gene/7157?source=CURATED&min_score=0.3&limit=10&format=json
+
+# Disease-gene for Breast Cancer (UMLS CUI C0006142)
+/gda/disease/C0006142?limit=10
+
+# Variant-disease for rs1042522
+/vda/variant/rs1042522
+```
+
+## Rate Limits
+Free academic tier: ~few hundred requests/day. Paid tiers available.
+
+## Free alternative
+If no API key: use **Open Targets** for disease-gene associations.

package/skills/database-lookup/references/drugbank.md

@@ -0,0 +1,54 @@
+# DrugBank API
+
+## Important: DrugBank's full API is commercial (paid license required)
+
+**Free alternatives for drug data:**
+- **ChEMBL** — extensive bioactivity data, free API
+- **PubChem** — free compound data
+- **OpenFDA** — drug labels, adverse events
+- **DGIdb** (https://dgidb.org/api) — drug-gene interactions, free
+
+## Base URL (Paid API)
+```
+https://api.drugbank.com/v1
+```
+
+## Auth
+API key required: `Authorization: Bearer <api_key>`
+
+## Key Endpoints (Paid API)
+
+| Endpoint | Description |
+|----------|-------------|
+| `/drugs/{drugbank_id}` | Get drug by DrugBank ID |
+| `/drugs?q={query}` | Search drugs |
+| `/drugs/{id}/interactions` | Drug-drug interactions |
+| `/drugs/{id}/targets` | Drug targets |
+| `/drugs/{id}/enzymes` | Metabolizing enzymes |
+| `/drugs/{id}/pathways` | Associated pathways |
+| `/drugs/{id}/adverse_effects` | Adverse effects |
+| `/drug_interactions?drugbank_id={id1},{id2}` | Check specific interactions |
+
+## Example Calls
+```
+GET /drugs/DB00945  (aspirin)
+GET /drugs?q=aspirin
+GET /drugs/DB00945/interactions
+GET /drugs/DB00945/targets
+```
+
+## Response Format
+```json
+{
+  "drugbank_id": "DB00945",
+  "name": "Acetylsalicylic acid",
+  "cas_number": "50-78-2",
+  "groups": ["approved"],
+  "targets": [{"name": "Prostaglandin G/H synthase 1", "uniprot_id": "P23219", "gene_name": "PTGS1", "actions": ["inhibitor"]}],
+  "external_ids": {"chembl": "CHEMBL25", "pubchem_compound": "2244"}
+}
+```
+
+## Free Access Options
+- **DrugBank Open Data**: ~2,500 FDA-approved drugs as XML/CSV download from https://go.drugbank.com/releases/latest
+- **Academic License**: Free for non-commercial use, provides data downloads (not API)

package/skills/database-lookup/references/ecb.md

@@ -0,0 +1,191 @@
+# ECB Statistical Data Warehouse (SDW) REST API Reference
+
+## Overview
+The ECB SDW API provides access to European Central Bank statistical data: exchange rates, monetary aggregates, interest rates, balance of payments, banking statistics, and more. It follows the SDMX (Statistical Data and Metadata eXchange) RESTful web services standard.
+
+## Base URL
+```
+https://data-api.ecb.europa.eu/service
+```
+
+Note: The legacy URL `https://sdw-wsrest.ecb.europa.eu/service` still works but the above is the current endpoint.
+
+## Authentication
+**No API key required.** The API is fully open and public.
+
+## Rate Limits
+- No formal rate limits published.
+- ECB asks users to be respectful: avoid excessive parallel requests.
+- For bulk downloads, use compressed responses (`Accept-Encoding: gzip`).
+
+## Common Headers
+| Header | Value | Description |
+|--------|-------|-------------|
+| `Accept` | `application/vnd.sdmx.data+json;version=2.0.0` | JSON format (recommended) |
+| `Accept` | `application/vnd.sdmx.data+csv` | CSV format |
+| `Accept` | `application/vnd.sdmx.data+xml` | SDMX-ML XML (default) |
+| `Accept-Encoding` | `gzip` | Compressed response |
+
+---
+
+## Key Endpoints
+
+### 1. Get Data (Time Series)
+
+```
+GET /data/{flowRef}/{key}?{parameters}
+```
+
+| Component | Description |
+|-----------|-------------|
+| `flowRef` | Dataflow ID (e.g., `EXR` for exchange rates, `BSI` for balance sheet items) |
+| `key` | Dot-separated dimension values. Use `+` for OR, `.` to skip a dimension (wildcard). |
+
+**Query Parameters:**
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `startPeriod` | No | Start date: `YYYY`, `YYYY-MM`, or `YYYY-MM-DD` |
+| `endPeriod` | No | End date: same formats |
+| `updatedAfter` | No | ISO 8601 timestamp; returns only data updated after this time |
+| `detail` | No | `full` (default), `dataonly`, `serieskeysonly`, `nodata` |
+| `firstNObservations` | No | Return only first N observations per series |
+| `lastNObservations` | No | Return only last N observations per series |
+| `dimensionAtObservation` | No | Typically `TIME_PERIOD` (default) |
+
+**Exchange Rate Key Structure (EXR dataflow):**
+`{frequency}.{currency}.{currency_denom}.{exr_type}.{exr_suffix}`
+
+| Position | Dimension | Common Values |
+|----------|-----------|---------------|
+| 1 | Frequency | `D` (daily), `M` (monthly), `A` (annual) |
+| 2 | Currency | `USD`, `GBP`, `JPY`, `CHF`, `CNY`, etc. |
+| 3 | Currency denominator | `EUR` (usually) |
+| 4 | Exchange rate type | `SP00` (spot), `EN00` (average) |
+| 5 | Exchange rate suffix | `A` (average), `E` (end of period) |
+
+**Example -- Daily USD/EUR spot rate, 2024:**
+```
+GET https://data-api.ecb.europa.eu/service/data/EXR/D.USD.EUR.SP00.A?startPeriod=2024-01-01&endPeriod=2024-12-31
+Accept: application/vnd.sdmx.data+json;version=2.0.0
+```
+
+**Example -- Monthly GBP and JPY vs EUR, last 12 observations:**
+```
+GET https://data-api.ecb.europa.eu/service/data/EXR/M.GBP+JPY.EUR.SP00.A?lastNObservations=12
+Accept: application/vnd.sdmx.data+json;version=2.0.0
+```
+
+**Example -- All daily exchange rates for a specific date (wildcard):**
+```
+GET https://data-api.ecb.europa.eu/service/data/EXR/D..EUR.SP00.A?startPeriod=2024-06-01&endPeriod=2024-06-01
+Accept: application/vnd.sdmx.data+json;version=2.0.0
+```
+
+**JSON Response Structure (SDMX-JSON v2.0):**
+```json
+{
+  "meta": { "schema": "...", "id": "...", "prepared": "2024-11-01T12:00:00Z" },
+  "data": {
+    "dataSets": [
+      {
+        "action": "Information",
+        "series": {
+          "0": {
+            "attributes": [0, 0, ...],
+            "observations": {
+              "0": [1.0856],
+              "1": [1.0791],
+              "2": [1.0834]
+            }
+          }
+        }
+      }
+    ],
+    "structures": [
+      {
+        "dimensions": {
+          "series": [...],
+          "observation": [
+            {
+              "id": "TIME_PERIOD",
+              "values": [
+                {"id": "2024-01-02", "name": "2024-01-02"},
+                {"id": "2024-01-03", "name": "2024-01-03"}
+              ]
+            }
+          ]
+        }
+      }
+    ]
+  }
+}
+```
+
+Note: Observation values are indexed arrays. Match observation index to `TIME_PERIOD` values in `structures.dimensions.observation`.
+
+**CSV Response** (simpler to parse):
+```
+Accept: application/vnd.sdmx.data+csv
+```
+Returns standard CSV with columns: `DATAFLOW`, `FREQ`, `CURRENCY`, `CURRENCY_DENOM`, `EXR_TYPE`, `EXR_SUFFIX`, `TIME_PERIOD`, `OBS_VALUE`, etc.
+
+---
+
+### 2. Get Dataflow Definitions (Available Datasets)
+
+```
+GET /dataflow/{agencyID}/{resourceID}/{version}
+```
+
+**Example -- List all ECB dataflows:**
+```
+GET https://data-api.ecb.europa.eu/service/dataflow/ECB
+Accept: application/vnd.sdmx.structure+json;version=2.0.0
+```
+
+**Example -- Get EXR dataflow definition:**
+```
+GET https://data-api.ecb.europa.eu/service/dataflow/ECB/EXR
+Accept: application/vnd.sdmx.structure+json;version=2.0.0
+```
+
+---
+
+### 3. Get Data Structure Definition (Dimensions & Codes)
+
+```
+GET /datastructure/{agencyID}/{resourceID}/{version}?references=children
+```
+
+**Example:**
+```
+GET https://data-api.ecb.europa.eu/service/datastructure/ECB/ECB_EXR1?references=children
+Accept: application/vnd.sdmx.structure+json;version=2.0.0
+```
+
+This returns all dimensions, their code lists, and allowed values -- essential for constructing valid keys.
+
+---
+
+## Common Dataflow IDs
+
+| Dataflow | Description |
+|----------|-------------|
+| `EXR` | Exchange rates |
+| `BSI` | Balance sheet items (monetary financial institutions) |
+| `MIR` | MFI interest rates |
+| `ILM` | Internal liquidity management |
+| `SEC` | Securities issues statistics |
+| `BOP` | Balance of payments |
+| `STP` | Structural financial indicators |
+| `CBD` | Consolidated banking data |
+| `ICP` | Index of consumer prices (HICP) |
+| `FM` | Financial market data |
+| `YC` | Yield curve data |
+
+## Notes
+- The SDMX-JSON format is verbose. For simpler parsing, use `Accept: application/vnd.sdmx.data+csv`.
+- When a dimension is unknown, leave it empty (e.g., `D..EUR.SP00.A`) to get all values for that dimension.
+- Use `+` to request multiple values for one dimension (e.g., `USD+GBP`).
+- The `detail=dataonly` parameter omits attributes and reduces response size.
+- Historical data availability varies by dataflow; exchange rates go back to 1999 (euro introduction).