clearai-dsh 0.1.0

package/presets/clearai/template/skills/paper-lookup/references/arxiv.md

@@ -0,0 +1,161 @@
+# arXiv API
+
+arXiv is a preprint server for physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering, and economics.
+
+**Important:** The arXiv API returns **Atom XML**, not JSON. There is no JSON option.
+
+## Base URL
+
+```
+https://export.arxiv.org/api/query
+```
+
+## Authentication
+
+None required. Fully public.
+
+## Query Parameters
+
+```
+GET https://export.arxiv.org/api/query?search_query={query}&start={n}&max_results={n}
+```
+
+| Parameter | Required | Default | Description |
+|-----------|----------|---------|-------------|
+| `search_query` | Yes* | -- | Search using field prefixes + boolean operators |
+| `id_list` | Yes* | -- | Comma-separated arXiv IDs (e.g., `2103.15348,2005.14165`) |
+| `start` | No | 0 | Pagination offset (0-based) |
+| `max_results` | No | 10 | Results per request (max 2000; absolute max 30000) |
+| `sortBy` | No | `relevance` | `relevance`, `lastUpdatedDate`, `submittedDate` |
+| `sortOrder` | No | `descending` | `ascending` or `descending` |
+
+*At least one of `search_query` or `id_list` must be provided. They can be combined (intersection).
+
+## Search Field Prefixes
+
+| Prefix | Searches |
+|--------|----------|
+| `ti:` | Title |
+| `au:` | Author |
+| `abs:` | Abstract |
+| `co:` | Comment |
+| `jr:` | Journal reference |
+| `cat:` | Subject category |
+| `rn:` | Report number |
+| `all:` | All fields |
+
+## Boolean Operators
+
+- `AND` -- both conditions
+- `OR` -- either condition
+- `ANDNOT` -- exclude
+- Parentheses for grouping (URL-encode as `%28` / `%29`)
+- Quoted phrases (URL-encode as `%22`)
+
+## Example Queries
+
+**Search all fields:**
+```
+https://export.arxiv.org/api/query?search_query=all:transformer+attention&max_results=5
+```
+
+**Author + category:**
+```
+https://export.arxiv.org/api/query?search_query=au:hinton+AND+cat:cs.LG&max_results=10
+```
+
+**Title search:**
+```
+https://export.arxiv.org/api/query?search_query=ti:%22attention+is+all+you+need%22
+```
+
+**By ID:**
+```
+https://export.arxiv.org/api/query?id_list=2103.15348
+```
+
+**Multiple IDs:**
+```
+https://export.arxiv.org/api/query?id_list=2103.15348,2005.14165,1706.03762
+```
+
+**Date range:**
+```
+https://export.arxiv.org/api/query?search_query=cat:cs.AI+AND+submittedDate:[202401010000+TO+202412312359]
+```
+
+## Response Format (Atom XML)
+
+```xml
+<feed xmlns="http://www.w3.org/2005/Atom">
+  <opensearch:totalResults>1234</opensearch:totalResults>
+  <opensearch:startIndex>0</opensearch:startIndex>
+  <opensearch:itemsPerPage>10</opensearch:itemsPerPage>
+
+  <entry>
+    <id>http://arxiv.org/abs/1706.03762v7</id>
+    <title>Attention Is All You Need</title>
+    <summary>The dominant sequence transduction models are based on...</summary>
+    <published>2017-06-12T17:57:34Z</published>
+    <updated>2023-08-02T00:00:12Z</updated>
+    <author><name>Ashish Vaswani</name></author>
+    <author><name>Noam Shazeer</name></author>
+    <!-- more authors -->
+    <category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
+    <arxiv:primary_category term="cs.CL"/>
+    <link rel="alternate" href="http://arxiv.org/abs/1706.03762v7"/>
+    <link rel="related" type="application/pdf" href="http://arxiv.org/pdf/1706.03762v7"/>
+    <arxiv:doi>10.48550/arXiv.1706.03762</arxiv:doi>
+    <arxiv:comment>15 pages, 5 figures</arxiv:comment>
+    <arxiv:journal_ref>Advances in Neural Information Processing Systems 30 (NIPS 2017)</arxiv:journal_ref>
+  </entry>
+</feed>
+```
+
+### Key XML elements per entry
+
+| Element | Description |
+|---------|-------------|
+| `<id>` | arXiv URL: `http://arxiv.org/abs/{id}` |
+| `<title>` | Paper title |
+| `<summary>` | Abstract |
+| `<published>` | Original submission date (ISO 8601) |
+| `<updated>` | Date of latest version |
+| `<author><name>` | One per author |
+| `<category term="...">` | Subject categories |
+| `<arxiv:primary_category>` | Primary classification |
+| `<link rel="alternate">` | Abstract page URL |
+| `<link rel="related" title="pdf">` | PDF URL |
+| `<arxiv:doi>` | DOI (when available) |
+| `<arxiv:comment>` | Author comments |
+| `<arxiv:journal_ref>` | Journal reference |
+
+## Parsing Tips
+
+Since arXiv returns XML, you'll need to parse it. With `curl`, you can pipe the output and extract what you need. The XML namespace is `http://www.w3.org/2005/Atom` with arXiv extensions in `http://arxiv.org/schemas/atom`.
+
+For practical extraction, the key data is in `<entry>` elements. Each entry's `<id>` contains the arXiv ID in the URL path.
+
+## Common Categories
+
+| Category | Field |
+|----------|-------|
+| `cs.AI` | Artificial Intelligence |
+| `cs.CL` | Computation and Language (NLP) |
+| `cs.CV` | Computer Vision |
+| `cs.LG` | Machine Learning |
+| `stat.ML` | Machine Learning (Statistics) |
+| `q-bio` | Quantitative Biology |
+| `physics` | Physics (all subcategories) |
+| `math` | Mathematics (all subcategories) |
+| `econ` | Economics |
+| `eess` | Electrical Engineering and Systems Science |
+
+Full list: https://arxiv.org/category_taxonomy
+
+## Rate Limits
+
+- **1 request every 3 seconds** (hard limit)
+- Single connection at a time
+- Search results are cached daily -- same query won't show new results within 24 hours
+- For bulk data, use the OAI-PMH interface instead

package/presets/clearai/template/skills/paper-lookup/references/biorxiv.md

@@ -0,0 +1,118 @@
+# bioRxiv API
+
+bioRxiv is a preprint server for biology. The API provides metadata for preprints, including title, authors, abstract, DOI, and publication status.
+
+**Important:** The bioRxiv API has **no keyword search**. It supports date-range browsing and DOI lookup only. For keyword search of bioRxiv preprints, use Semantic Scholar, OpenAlex, or CORE instead.
+
+## Base URL
+
+```
+https://api.biorxiv.org
+```
+
+## Authentication
+
+None required. Fully public API.
+
+## Key Endpoints
+
+### 1. Content Detail -- Browse by date range
+
+```
+GET /details/biorxiv/{interval}/{cursor}/{format}
+```
+
+| Parameter | Values | Description |
+|-----------|--------|-------------|
+| `interval` | `YYYY-MM-DD/YYYY-MM-DD` | Date range (inclusive). Keep ranges narrow (1-3 days) to avoid timeouts. |
+| | `N` (integer) | N most recent preprints |
+| | `Nd` (integer + "d") | Last N days |
+| `cursor` | Integer (default `0`) | Pagination offset (100 results per page) |
+| `format` | `json` (default), `xml` | Response format |
+
+Optional query parameter: `?category=neuroscience` (filter by category, use underscores for spaces)
+
+**Examples:**
+```
+https://api.biorxiv.org/details/biorxiv/2024-01-01/2024-01-31/0
+https://api.biorxiv.org/details/biorxiv/5
+https://api.biorxiv.org/details/biorxiv/10d
+https://api.biorxiv.org/details/biorxiv/2024-01-01/2024-01-31?category=neuroscience
+```
+
+### 2. Content Detail -- DOI lookup
+
+```
+GET /details/biorxiv/{doi}/na/{format}
+```
+
+**Example:**
+```
+https://api.biorxiv.org/details/biorxiv/10.1101/2024.01.16.575895/na/json
+```
+
+### 3. Published Article Links
+
+```
+GET /pubs/biorxiv/{interval}/{cursor}
+GET /pubs/biorxiv/{doi}/na
+```
+
+Links preprints to their published journal versions. Accepts both preprint DOI and published DOI.
+
+### 4. Publisher Filter
+
+```
+GET /publisher/{prefix}/{interval}/{cursor}
+```
+
+Find bioRxiv papers published by a specific publisher (by DOI prefix).
+
+**Example:**
+```
+https://api.biorxiv.org/publisher/10.15252/2024-01-01/2024-06-01/0
+```
+
+## Response Format
+
+```json
+{
+  "messages": [{
+    "status": "ok",
+    "count": 100,
+    "total": "1029",
+    "cursor": 0
+  }],
+  "collection": [{
+    "title": "Paper title...",
+    "authors": "Surname, A.; Surname, B.",
+    "author_corresponding": "Full Name",
+    "author_corresponding_institution": "Institution",
+    "doi": "10.1101/2024.01.16.575895",
+    "date": "2024-01-20",
+    "version": "1",
+    "type": "new results",
+    "license": "cc_no",
+    "category": "cancer biology",
+    "jatsxml": "https://www.biorxiv.org/content/early/.../source.xml",
+    "abstract": "Full abstract text...",
+    "published": "10.1158/2159-8290.CD-24-0187",
+    "server": "bioRxiv"
+  }]
+}
+```
+
+- `published` is `"NA"` if not yet published in a journal, or the published DOI if it has been.
+- `type` values: `new results`, `confirmatory results`, `contradictory results`
+
+## Pagination
+
+All multi-result endpoints return **100 results per page**. Use `cursor` to paginate. The `messages` object tells you the `total` count.
+
+## Rate Limits
+
+No documented rate limits. No authentication required. Be reasonable with request frequency.
+
+## Categories
+
+`animal-behavior-and-cognition`, `biochemistry`, `bioengineering`, `bioinformatics`, `biophysics`, `cancer-biology`, `cell-biology`, `clinical-trials`, `developmental-biology`, `ecology`, `epidemiology`, `evolutionary-biology`, `genetics`, `genomics`, `immunology`, `microbiology`, `molecular-biology`, `neuroscience`, `paleontology`, `pathology`, `pharmacology-and-toxicology`, `physiology`, `plant-biology`, `scientific-communication-and-education`, `synthetic-biology`, `systems-biology`, `zoology`

package/presets/clearai/template/skills/paper-lookup/references/core.md

@@ -0,0 +1,150 @@
+# CORE API
+
+CORE aggregates open access research from 15,000+ repositories worldwide. It provides **full text** for 37M+ articles and metadata for 368M+ papers.
+
+## Base URL
+
+```
+https://api.core.ac.uk/v3
+```
+
+**Important:** GET search paths require a **trailing slash** (e.g., `/v3/search/works/` not `/v3/search/works`).
+
+## Authentication
+
+- **Header:** `Authorization: Bearer YOUR_API_KEY`
+- **Query param:** `?api_key=YOUR_API_KEY`
+- Register at: https://core.ac.uk/services/api
+
+**Without auth:** Basic metadata queries work, but full text is NOT available (returns "Not available for public API users").
+
+## Rate Limits (token-based)
+
+| User Type | Daily Tokens | Per-Minute Max |
+|-----------|-------------|----------------|
+| Unauthenticated | 100/day | 10/min |
+| Registered Personal | 1,000/day | 25/min |
+| Registered Academic | 5,000/day | 10/min |
+
+Simple queries cost 1 token. Downloads and scroll pagination cost 3-5 tokens.
+
+## Key Endpoints
+
+### 1. Search works
+
+```
+GET /v3/search/works/?q={query}&limit={n}&offset={n}
+```
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `q` | required | Search query (supports field lookups, boolean operators) |
+| `limit` | 10 | Results per page (max 100) |
+| `offset` | 0 | Pagination offset |
+| `scroll` | false | Enable scroll pagination for >10,000 results |
+| `sort` | relevance | `relevance` or `recency` |
+
+**POST alternative** (for complex queries):
+```
+POST /v3/search/works
+Content-Type: application/json
+
+{"q": "machine learning", "limit": 10, "offset": 0}
+```
+
+**Example:**
+```
+https://api.core.ac.uk/v3/search/works/?q=CRISPR+gene+therapy&limit=10
+```
+
+### 2. Query language
+
+| Operator | Example | Description |
+|----------|---------|-------------|
+| AND | `title:"AI" AND authors:"Smith"` | Both conditions |
+| OR | `title:"AI" OR fullText:"Deep Learning"` | Either condition |
+| Grouping | `(title:"AI" OR title:"ML") AND yearPublished>"2020"` | Precedence |
+| Field lookup | `title:"Machine Learning"` | Search specific field |
+| Range | `yearPublished>2018` | Numeric comparison |
+| Exists | `_exists_:fullText` | Field must exist |
+| Phrase | `title:"Attention is all you need"` | Exact phrase |
+
+**Searchable fields:** `abstract`, `arxivId`, `authors`, `contributors`, `createdDate`, `dataProviders`, `depositedDate`, `documentType`, `doi`, `fullText`, `id`, `language`, `license`, `oai`, `title`, `yearPublished`
+
+### 3. Get work by ID
+
+```
+GET /v3/works/{id}
+```
+
+`id` is a CORE Work ID (integer). Example: `/v3/works/267312`
+
+### 4. Get output by ID
+
+```
+GET /v3/outputs/{id}
+```
+
+### 5. Download full text
+
+```
+GET /v3/outputs/{id}/download
+```
+
+Returns binary PDF. Requires authentication.
+
+```
+GET /v3/works/tei/{id}
+```
+
+Returns TEI XML format.
+
+### 6. Search outputs
+
+```
+GET /v3/search/outputs/?q={query}&limit={n}&offset={n}
+```
+
+Search by DOI: `q=doi:10.1038/nature12373`
+
+## Response Format
+
+### Search response
+```json
+{
+  "totalHits": 2281337,
+  "limit": 10,
+  "offset": 0,
+  "scrollId": null,
+  "results": [...]
+}
+```
+
+### Work object (key fields)
+```json
+{
+  "id": 8848131,
+  "title": "Attention Is All You Need",
+  "authors": [{"name": "Ashish Vaswani"}, ...],
+  "abstract": "The dominant sequence...",
+  "doi": "10.48550/arXiv.1706.03762",
+  "arxivId": "1706.03762",
+  "yearPublished": 2017,
+  "downloadUrl": "https://core.ac.uk/download/...",
+  "fullText": "Full text content (when authenticated)...",
+  "language": {"code": "en", "name": "English"},
+  "documentType": "research",
+  "citationCount": 145678,
+  "dataProviders": [{"name": "arXiv"}],
+  "links": [{"type": "download", "url": "..."}]
+}
+```
+
+## Pagination
+
+- **Standard:** `offset` + `limit` (max 10,000 results)
+- **Scroll:** Set `scroll=true`. Response includes `scrollId`. Use in subsequent requests to page beyond 10,000 (costs more tokens).
+
+## Error Handling
+
+Under heavy load, the API may return partial shard failure messages. These are transient -- retry after a brief wait.

package/presets/clearai/template/skills/paper-lookup/references/crossref.md

@@ -0,0 +1,181 @@
+# Crossref API
+
+Crossref is the DOI registration agency for scholarly content. It provides metadata for 150M+ works including journal articles, books, conference papers, datasets, and preprints.
+
+## Base URL
+
+```
+https://api.crossref.org
+```
+
+## Authentication
+
+None required. Add `mailto=you@example.com` to get into the **polite pool** (2x faster rate limits).
+
+## Rate Limits
+
+| Pool | Rate | Concurrency |
+|------|------|-------------|
+| Public (no mailto) | 5 req/sec | 1 concurrent |
+| Polite (with mailto) | 10 req/sec | 3 concurrent |
+
+HTTP 429 = temporarily blocked.
+
+## Key Endpoints
+
+### 1. Search works
+
+```
+GET /works?query={text}&rows={n}&mailto=you@example.com
+```
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `query` | -- | Free-text search across all fields |
+| `query.author` | -- | Search author names |
+| `query.bibliographic` | -- | Search titles, authors, ISSNs, years |
+| `query.affiliation` | -- | Search affiliations |
+| `query.container-title` | -- | Search journal names |
+| `filter` | -- | Comma-separated `name:value` pairs |
+| `sort` | `score` | `score`, `published`, `issued`, `deposited`, `updated`, `is-referenced-by-count`, `references-count` |
+| `order` | `desc` | `asc` or `desc` |
+| `rows` | 20 | Results per page (max 1000) |
+| `offset` | 0 | Skip N results (max 10,000) |
+| `cursor` | -- | Use `*` for cursor-based deep pagination |
+| `select` | -- | Comma-separated field names to return |
+| `facet` | -- | Facet counts, e.g. `type-name:10` |
+| `sample` | -- | Return N random items (max 100) |
+
+**Example:**
+```
+https://api.crossref.org/works?query=CRISPR+gene+therapy&filter=from-pub-date:2024-01-01,type:journal-article,has-abstract:true&rows=5&sort=published&order=desc&mailto=you@example.com
+```
+
+### 2. Get work by DOI
+
+```
+GET /works/{doi}?mailto=you@example.com
+```
+
+URL-encode the DOI: `10.1038/nature12373` becomes `10.1038%2Fnature12373`
+
+**Example:**
+```
+https://api.crossref.org/works/10.1038%2Fnature12373?mailto=you@example.com
+```
+
+### 3. Journals
+
+```
+GET /journals?query={name}&rows={n}
+GET /journals/{issn}
+GET /journals/{issn}/works?query={text}&rows={n}
+```
+
+### 4. Funders
+
+```
+GET /funders?query={name}
+GET /funders/{id}
+GET /funders/{id}/works?rows={n}
+```
+
+Funder IDs are from the Funder Registry (e.g., `100000001` for NSF).
+
+### 5. Members (publishers)
+
+```
+GET /members?query={name}
+GET /members/{id}/works?rows={n}
+```
+
+## Key Filters
+
+### Date filters (accept `YYYY`, `YYYY-MM`, `YYYY-MM-DD`)
+| Filter | Description |
+|--------|-------------|
+| `from-pub-date` / `until-pub-date` | Publication date |
+| `from-print-pub-date` / `until-print-pub-date` | Print publication date |
+| `from-online-pub-date` / `until-online-pub-date` | Online publication date |
+| `from-posted-date` / `until-posted-date` | Posted date (preprints) |
+
+### Boolean filters
+| Filter | Description |
+|--------|-------------|
+| `has-abstract` | Has an abstract |
+| `has-orcid` | Has ORCID IDs |
+| `has-funder` | Has funder info |
+| `has-full-text` | Has full-text links |
+| `has-references` | Has reference list |
+| `has-license` | Has license info |
+
+### Value filters
+| Filter | Description |
+|--------|-------------|
+| `type` | `journal-article`, `posted-content`, `book-chapter`, `proceedings-article`, etc. |
+| `issn` | Journal ISSN |
+| `doi` | Specific DOI |
+| `orcid` | Contributor ORCID |
+| `funder` | Funder Registry ID |
+| `member` | Crossref member ID |
+| `prefix` | DOI prefix |
+| `license.url` | License URL |
+| `update-type` | `correction`, `retraction` |
+
+**Syntax:** `filter=name1:value1,name2:value2`
+
+## Pagination
+
+### Offset-based (max 10,000)
+```
+/works?query=cancer&rows=100&offset=200
+```
+
+### Cursor-based (unlimited)
+1. First request: `?cursor=*&rows=100`
+2. Response includes `next-cursor`
+3. Next request: `?cursor={next-cursor-value}&rows=100`
+4. Cursors expire after 5 minutes
+
+## Response Format
+
+### List response
+```json
+{
+  "status": "ok",
+  "message-type": "work-list",
+  "message": {
+    "total-results": 2779116,
+    "items-per-page": 20,
+    "next-cursor": "...",
+    "items": [...]
+  }
+}
+```
+
+### Work object (key fields)
+```json
+{
+  "DOI": "10.1038/nature12373",
+  "title": ["Nanometre-scale thermometry in a living cell"],
+  "author": [{"given": "G.", "family": "Kucsko", "sequence": "first"}],
+  "publisher": "Springer Science and Business Media LLC",
+  "type": "journal-article",
+  "published": {"date-parts": [[2013, 7, 31]]},
+  "container-title": ["Nature"],
+  "ISSN": ["0028-0836", "1476-4687"],
+  "volume": "500",
+  "issue": "7460",
+  "page": "54-58",
+  "is-referenced-by-count": 1745,
+  "references-count": 30,
+  "abstract": "<p>Abstract text with HTML tags...</p>",
+  "license": [{"URL": "...", "content-version": "vor"}],
+  "link": [{"URL": "...", "content-type": "application/pdf"}],
+  "reference": [{"key": "...", "doi-asserted-by": "crossref", "DOI": "..."}],
+  "subject": ["Multidisciplinary"],
+  "language": "en"
+}
+```
+
+Note: `title` and `container-title` are arrays. `published.date-parts` is `[[year, month, day]]`. Abstract may contain HTML tags.