@anolilab/ai-model-registry 1.3.0 → 2.0.0

package/README.md

@@ -1,51 +1,69 @@
-# @anolilab/ai-model-registry
+<!-- START_PACKAGE_OG_IMAGE_PLACEHOLDER -->
 
-[![npm version](https://badge.fury.io/js/@anolilab%2Fai-model-registry.svg)](https://badge.fury.io/js/@anolilab%2Fai-model-registry)
-[![License: Apache-2.0](https://img.shields.io/badge/License-Apache--2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
-[![TypeScript](https://img.shields.io/badge/TypeScript-5.8+-blue.svg)](https://www.typescriptlang.org/)
+<a href="https://www.anolilab.com/open-source" align="center">
 
-A comprehensive, unified registry for AI model providers and their metadata. This package provides a tree-shakable interface to access model information from 50+ AI providers including OpenAI, Anthropic, Google, Meta, Groq, and many others.
+  <img src="__assets__/package-og.svg" alt="ai-model-registry" />
 
-## ✨ Features
+</a>
 
-- **🔄 Unified Interface**: Access models from multiple providers through a single API
-- **🛡️ Type Safety**: Full TypeScript support with Zod schema validation
-- **📦 Tree Shaking**: Import only what you need to minimize bundle size
-- **💰 Rich Metadata**: Comprehensive model information including capabilities, pricing, and limits
-- **🔍 Powerful Search**: Advanced search and filtering capabilities across all models
-- **🔄 Auto-Sync**: Automatic data synchronization between models with the same ID
-- **💸 Pricing Integration**: Real-time pricing data from Helicone API (840+ models)
-- **📊 Provider Stats**: Detailed statistics and analytics
+<h3 align="center">Unified registry for AI model providers and their metadata</h3>
 
-## 📦 Installation
+<!-- END_PACKAGE_OG_IMAGE_PLACEHOLDER -->
 
-```bash
-# Using npm
+<br />
+
+<div align="center">
+
+[![typescript-image][typescript-badge]][typescript-url]
+[![apache licence][license-badge]][license]
+[![npm downloads][npm-downloads-badge]][npm-downloads]
+[![Chat][chat-badge]][chat]
+[![PRs Welcome][prs-welcome-badge]][prs-welcome]
+
+</div>
+
+---
+
+<div align="center">
+    <p>
+        <sup>
+            Daniel Bannert's open source work is supported by the community on <a href="https://github.com/sponsors/prisis">GitHub Sponsors</a>
+        </sup>
+    </p>
+</div>
+
+---
+
+## Install
+
+```sh
 npm install @anolilab/ai-model-registry
+```
 
-# Using yarn
+```sh
 yarn add @anolilab/ai-model-registry
+```
 
-# Using pnpm
+```sh
 pnpm add @anolilab/ai-model-registry
 ```
 
-## 🚀 Quick Start
+## Usage
 
 ```typescript
-import { getProviders, getModelsByProvider, getModelById, searchModels, getAllModels } from "@anolilab/ai-model-registry";
+import { getProviders, getModelsByProvider, getModelById, searchModels, getAllModels, type ProviderName } from "@anolilab/ai-model-registry";
 
-// Get all available providers
-const providers = getProviders();
+// Get all available providers (async)
+const providers = await getProviders();
 console.log(providers);
 // ['Anthropic', 'Google', 'Groq', 'Meta', 'OpenAI', 'DeepSeek', ...]
 
-// Get all models from a specific provider
-const anthropicModels = getModelsByProvider("Anthropic");
+// Get all models from a specific provider (type-safe provider name)
+const anthropicModels = await getModelsByProvider("Anthropic" as ProviderName);
 console.log(`Found ${anthropicModels.length} Anthropic models`);
 
 // Get a specific model by ID
-const model = getModelById("claude-3-opus-20240229");
+const model = await getModelById("claude-3-opus-20240229");
 if (model) {
     console.log(`Model: ${model.name}`);
     console.log(`Provider: ${model.provider}`);
@@ -54,60 +72,95 @@ if (model) {
 }
 
 // Search for models with specific capabilities
-const visionModels = searchModels({ vision: true });
-const reasoningModels = searchModels({ reasoning: true });
-const toolCallModels = searchModels({ tool_call: true });
+const visionModels = await searchModels({ vision: true });
+const reasoningModels = await searchModels({ reasoning: true });
+const toolCallModels = await searchModels({ tool_call: true });
 
 // Get all models for advanced filtering
-const allModels = getAllModels();
+const allModels = await getAllModels();
+```
+
+### Type-Safe Provider and Model Names
+
+The package provides TypeScript types for provider names and model IDs:
+
+```typescript
+import type { ProviderName } from "@anolilab/ai-model-registry";
+import type ModelName from "@anolilab/ai-model-registry/types/open-router";
+
+// Type-safe provider names
+const provider: ProviderName = "OpenAI"; // ✅ Valid
+const invalid: ProviderName = "InvalidProvider"; // ❌ TypeScript error
+
+// Type-safe model IDs for specific providers
+const modelId: ModelName = "meta-llama/llama-3.1-8b-instruct"; // ✅ Valid
+const invalidId: ModelName = "invalid-model"; // ❌ TypeScript error
 ```
 
-## 📚 API Reference
+## Features
+
+- Unified interface to access models from multiple providers through a single API
+- Full TypeScript support with Zod schema validation for type safety
+- Type-safe provider names and model IDs with generated union types
+- Dynamic imports for optimal code splitting and tree shaking
+- Provider-specific JSON files for efficient loading
+- Tree shaking support - import only what you need to minimize bundle size
+- Comprehensive model information including capabilities, pricing, and limits
+- Advanced search and filtering capabilities across all models
+- Automatic data synchronization between models with the same ID
+- Real-time pricing data integration from Helicone API (840+ models)
+- Detailed provider statistics and analytics
+
+## API Reference
 
 ### Core Functions
 
-#### `getProviders(): string[]`
+All functions are async and use dynamic imports for optimal code splitting.
+
+#### `getProviders(): Promise<ProviderName[]>`
 
 Returns an array of all available provider names.
 
 ```typescript
-const providers = getProviders();
+const providers = await getProviders();
 // ['Anthropic', 'Google', 'Groq', 'Meta', 'OpenAI', ...]
 ```
 
-#### `getModelsByProvider(provider: string): Model[]`
+#### `getModelsByProvider(provider: ProviderName): Promise<Model[]>`
 
-Returns all models for a specific provider.
+Returns all models for a specific provider. Uses provider-specific JSON files for efficient loading.
 
 ```typescript
-const openAIModels = getModelsByProvider("OpenAI");
-const anthropicModels = getModelsByProvider("Anthropic");
+import type { ProviderName } from "@anolilab/ai-model-registry";
+
+const openAIModels = await getModelsByProvider("OpenAI" as ProviderName);
+const anthropicModels = await getModelsByProvider("Anthropic" as ProviderName);
 ```
 
-#### `getModelById(id: string): Model | undefined`
+#### `getModelById(id: string): Promise<Model | undefined>`
 
 Returns a specific model by its ID, or `undefined` if not found.
 
 ```typescript
-const gpt4 = getModelById("gpt-4");
-const claude = getModelById("claude-3-opus-20240229");
+const gpt4 = await getModelById("gpt-4");
+const claude = await getModelById("claude-3-opus-20240229");
 ```
 
-#### `getAllModels(): Model[]`
+#### `getAllModels(): Promise<Model[]>`
 
-Returns all models (useful for advanced filtering and custom logic).
+Returns all models (useful for advanced filtering and custom logic). Loads all provider files in parallel.
 
 ```typescript
-const allModels = getAllModels();
+const allModels = await getAllModels();
 const expensiveModels = allModels.filter((model) => (model.cost.input || 0) > 0.1 || (model.cost.output || 0) > 0.1);
 ```
 
-#### `getProviderStats(): Record<string, number>`
+#### `getProviderStats(): Promise<Record<ProviderName, number>>`
 
 Returns provider statistics with model counts.
 
 ```typescript
-const stats = getProviderStats();
+const stats = await getProviderStats();
 console.log(stats);
 // {
 //   'OpenAI': 15,
@@ -120,9 +173,9 @@ console.log(stats);
 
 ### Advanced Search
 
-#### `searchModels(criteria: SearchCriteria): Model[]`
+#### `searchModels(criteria: SearchCriteria): Promise<Model[]>`
 
-Search models by various criteria with powerful filtering options.
+Search models by various criteria with powerful filtering options. Automatically uses provider-specific files when a provider filter is specified.
 
 ```typescript
 interface SearchCriteria {
@@ -134,7 +187,7 @@ interface SearchCriteria {
     preview?: boolean;
 
     // Provider filter
-    provider?: string;
+    provider?: ProviderName;
 
     // Modality filters
     modalities?: {
@@ -156,41 +209,35 @@ interface SearchCriteria {
 
 ```typescript
 // Find all vision-capable models
-const visionModels = searchModels({ vision: true });
+const visionModels = await searchModels({ vision: true });
 
 // Find models with reasoning capabilities
-const reasoningModels = searchModels({ reasoning: true });
+const reasoningModels = await searchModels({ reasoning: true });
 
 // Find models that support tool calling
-const toolCallModels = searchModels({ tool_call: true });
+const toolCallModels = await searchModels({ tool_call: true });
 
-// Find models from a specific provider
-const openAIModels = searchModels({ provider: "OpenAI" });
+// Find models from a specific provider (uses provider-specific file)
+const openAIModels = await searchModels({ provider: "OpenAI" as ProviderName });
 
 // Find models with large context windows
-const largeContextModels = searchModels({ context_min: 100000 });
-
-// Find affordable models
-const affordableModels = searchModels({
-    max_input_cost: 0.01,
-    max_output_cost: 0.02,
-});
+const largeContextModels = await searchModels({ context_min: 100000 });
 
 // Find models that accept text and image input
-const multimodalModels = searchModels({
+const multimodalModels = await searchModels({
     modalities: {
         input: ["text", "image"],
     },
 });
 
 // Find models with streaming support
-const streamingModels = searchModels({ streaming_supported: true });
+const streamingModels = await searchModels({ streaming_supported: true });
 
 // Find preview/beta models
-const previewModels = searchModels({ preview: true });
+const previewModels = await searchModels({ preview: true });
 ```
 
-## 🏗️ Model Schema
+## Model Schema
 
 Each model follows a comprehensive schema with the following structure:
 
@@ -283,54 +330,154 @@ interface Model {
 }
 ```
 
-## 🌳 Tree Shaking
+## Tree Shaking and Code Splitting
 
-The package supports tree shaking, so you can import only what you need:
+The package supports tree shaking and uses dynamic imports for optimal code splitting:
 
 ```typescript
 // Only import specific functions
-import { getProviders, getModelById } from "@anolilab/ai-model-registry";
+import { getProviders, getModelById, type ProviderName } from "@anolilab/ai-model-registry";
 
 // Import schema for validation
 import { ModelSchema } from "@anolilab/ai-model-registry/schema";
 
 // Import icons (if needed)
 import { getIcon } from "@anolilab/ai-model-registry/icons";
+
+// Import provider-specific model ID types
+import type ModelName from "@anolilab/ai-model-registry/types/open-router";
+import type { ProviderName } from "@anolilab/ai-model-registry/types/providers";
 ```
 
-## 🏢 Supported Providers
+### Dynamic Loading
+
+The package uses dynamic imports to load provider-specific JSON files on demand:
+
+- **Single provider queries**: Only loads the specific provider's JSON file
+- **All models queries**: Loads all provider files in parallel
+- **Automatic caching**: Results are cached to avoid re-loading
+- **Better code splitting**: Bundlers can split provider files into separate chunks
+
+## Supported Providers
 
 The registry includes models from 50+ providers:
 
 ### Major Providers
 
-- **OpenAI** (GPT-4, GPT-3.5, O1, O3, etc.)
-- **Anthropic** (Claude 3.5, Claude 3, Claude 2.1, etc.)
-- **Google** (Gemini 2.5, Gemini 1.5, PaLM, etc.)
-- **Meta** (Llama 3, Llama 2, Code Llama, etc.)
-- **Groq** (Various models with ultra-fast inference)
-- **DeepSeek** (DeepSeek R1, DeepSeek V3, etc.)
+- OpenAI (GPT-4, GPT-3.5, O1, O3, etc.)
+- Anthropic (Claude 3.5, Claude 3, Claude 2.1, etc.)
+- Google (Gemini 2.5, Gemini 1.5, PaLM, etc.)
+- Meta (Llama 3, Llama 2, Code Llama, etc.)
+- Groq (Various models with ultra-fast inference)
+- DeepSeek (DeepSeek R1, DeepSeek V3, etc.)
 
 ### Specialized Providers
 
-- **Mistral AI** (Mistral Large, Mixtral, etc.)
-- **Cohere** (Command R, Command A, etc.)
-- **Perplexity** (Sonar, Sonar Pro, etc.)
-- **Together AI** (Various open models)
-- **Fireworks AI** (Various models)
-- **Vercel** (v0 models)
+- Mistral AI (Mistral Large, Mixtral, etc.)
+- Cohere (Command R, Command A, etc.)
+- Perplexity (Sonar, Sonar Pro, etc.)
+- Together AI (Various open models)
+- Fireworks AI (Various models)
+- Vercel (v0 models)
 
 ### Open Source & Research
 
-- **HuggingFace** (Various hosted models)
-- **ModelScope** (Chinese models)
-- **OpenRouter** (Aggregated models)
-- **GitHub Copilot** (Code models)
-- **Azure** (OpenAI models)
+- HuggingFace (Various hosted models)
+- ModelScope (Chinese models)
+- OpenRouter (Aggregated models)
+- GitHub Copilot (Code models)
+- Azure (OpenAI models)
 
 And many more...
 
-## 🛠️ Development
+## Pricing Data Integration
+
+This package automatically includes real-time pricing data from [Helicone's LLM Cost API](https://helicone.ai/api/llm-costs) during the aggregation process.
+
+### Features
+
+- Automatic enrichment: Pricing data is automatically added during aggregation
+- Smart matching: Uses multiple strategies to match models with pricing data
+- Non-destructive: Preserves existing pricing data while filling in missing values
+- Cost conversion: Automatically converts from per 1M tokens to per 1K tokens format
+- 840+ models: Covers pricing for 840+ models across all major providers
+
+### Supported Pricing Providers
+
+Helicone provides pricing data for models from:
+
+- OpenAI (GPT-4, GPT-3.5, O1, O3, etc.)
+- Anthropic (Claude models)
+- Google (Gemini models)
+- Meta (Llama models)
+- Mistral (Mistral models)
+- Groq (Various models)
+- And many more...
+
+## Model Data Synchronization
+
+The provider registry includes a powerful data synchronization system that automatically merges missing data between models with the same ID across different providers.
+
+### How It Works
+
+1. Groups models by ID: Finds all models with the same ID across different providers
+2. Calculates completeness scores: Evaluates how complete each model's data is (excluding cost fields)
+3. Uses the most complete model as base: Selects the model with the highest data completeness
+4. Merges missing data: Fills in missing fields from other models with the same ID
+5. Preserves cost data: Never overwrites existing cost information
+
+### Protected Fields
+
+The following cost-related fields are never synchronized to preserve pricing accuracy:
+
+- `cost` (entire cost object)
+- `input` (input cost)
+- `output` (output cost)
+- `inputCacheHit` (cache hit pricing)
+- `imageGeneration` (image generation pricing)
+- `videoGeneration` (video generation pricing)
+
+### Example
+
+If you have the same model (e.g., `gpt-4`) from multiple providers:
+
+**OpenAI Provider:**
+
+```json
+{
+    "id": "gpt-4",
+    "name": "GPT-4",
+    "cost": { "input": 0.03, "output": 0.06 },
+    "description": null,
+    "releaseDate": "2023-03-14"
+}
+```
+
+**Azure Provider:**
+
+```json
+{
+    "id": "gpt-4",
+    "name": null,
+    "cost": { "input": 0.03, "output": 0.06 },
+    "description": "GPT-4 is a large multimodal model",
+    "releaseDate": null
+}
+```
+
+**Result after synchronization:**
+
+```json
+{
+    "id": "gpt-4",
+    "name": "GPT-4",
+    "cost": { "input": 0.03, "output": 0.06 },
+    "description": "GPT-4 is a large multimodal model",
+    "releaseDate": "2023-03-14"
+}
+```
+
+## Development
 
 ### Prerequisites
 
@@ -384,7 +531,11 @@ pnpm run build
     - Fetches pricing data from Helicone API
     - Enriches models with icon information
     - Synchronizes data between models with the same ID
-    - Generates `data/all-models.json` and `src/models-data.ts`
+    - Generates `public/api.json` (main API file)
+    - Generates `public/{provider-name}.json` (provider-specific files)
+    - Generates `public/providers.json` (provider index)
+    - Generates `src/types/providers.ts` (ProviderName type)
+    - Generates `src/types/{provider-name}.ts` (ModelName types per provider)
 
 3. **Generate Icons (`pnpm run generate-icons`)**:
     - Creates SVG sprite sheet from LobeHub icons and custom icons
@@ -436,120 +587,39 @@ pnpm run lint:types
 ```
 packages/ai-model-registry/
 ├── src/
-│   ├── index.ts          # Main exports
+│   ├── index.ts          # Main exports (async functions)
 │   ├── schema.ts         # Model schema definitions
-│   └── models-data.ts    # Generated model data
+│   ├── icons-sprite.ts   # Generated icon sprite
+│   └── types/            # Generated TypeScript types
+│       ├── providers.ts  # ProviderName type
+│       ├── open-router.ts  # ModelName type for OpenRouter
+│       ├── open-ai.ts    # ModelName type for OpenAI
+│       └── ...           # Other provider model types
 ├── scripts/
 │   ├── aggregate-providers.ts    # Data aggregation script
 │   ├── generate-svg-sprite.ts    # Icon generation
 │   └── download/                 # Provider data downloaders
+├── public/               # Generated JSON files
+│   ├── api.json          # Main API file (all models)
+│   ├── providers.json    # Provider index
+│   ├── open-router.json  # Provider-specific models
+│   ├── open-ai.json      # Provider-specific models
+│   └── ...               # Other provider files
 ├── data/
-│   ├── all-models.json   # Generated model data
 │   └── providers/        # Raw provider data
 └── assets/
     └── icons/            # Provider icons
 ```
 
-## 💰 Pricing Data Integration
-
-This package automatically includes real-time pricing data from [Helicone's LLM Cost API](https://helicone.ai/api/llm-costs) during the aggregation process.
-
-### Features
-
-- **🔄 Automatic Enrichment**: Pricing data is automatically added during aggregation
-- **🎯 Smart Matching**: Uses multiple strategies to match models with pricing data
-- **🛡️ Non-Destructive**: Preserves existing pricing data while filling in missing values
-- **🔄 Cost Conversion**: Automatically converts from per 1M tokens to per 1K tokens format
-- **📊 840+ Models**: Covers pricing for 840+ models across all major providers
-
-### Supported Pricing Providers
-
-Helicone provides pricing data for models from:
-
-- OpenAI (GPT-4, GPT-3.5, O1, O3, etc.)
-- Anthropic (Claude models)
-- Google (Gemini models)
-- Meta (Llama models)
-- Mistral (Mistral models)
-- Groq (Various models)
-- And many more...
-
-### Usage
-
-```bash
-# Aggregate all models with pricing data
-pnpm run aggregate
-
-# Build the package (includes aggregation with pricing)
-pnpm run build
-```
-
-## 🔄 Model Data Synchronization
-
-The provider registry includes a powerful data synchronization system that automatically merges missing data between models with the same ID across different providers.
-
-### How It Works
-
-1. **Groups models by ID**: Finds all models with the same ID across different providers
-2. **Calculates completeness scores**: Evaluates how complete each model's data is (excluding cost fields)
-3. **Uses the most complete model as base**: Selects the model with the highest data completeness
-4. **Merges missing data**: Fills in missing fields from other models with the same ID
-5. **Preserves cost data**: Never overwrites existing cost information
-
-### Protected Fields
-
-The following cost-related fields are **never synchronized** to preserve pricing accuracy:
-
-- `cost` (entire cost object)
-- `input` (input cost)
-- `output` (output cost)
-- `inputCacheHit` (cache hit pricing)
-- `imageGeneration` (image generation pricing)
-- `videoGeneration` (video generation pricing)
-
-### Example
-
-If you have the same model (e.g., `gpt-4`) from multiple providers:
-
-**OpenAI Provider:**
-
-```json
-{
-    "id": "gpt-4",
-    "name": "GPT-4",
-    "cost": { "input": 0.03, "output": 0.06 },
-    "description": null,
-    "releaseDate": "2023-03-14"
-}
-```
-
-**Azure Provider:**
+## Supported Node.js Versions
 
-```json
-{
-    "id": "gpt-4",
-    "name": null,
-    "cost": { "input": 0.03, "output": 0.06 },
-    "description": "GPT-4 is a large multimodal model",
-    "releaseDate": null
-}
-```
+Libraries in this ecosystem make the best effort to track [Node.js' release schedule](https://nodejs.org/en/about/releases/). Here's [a post on why we think this is important](https://medium.com/the-node-js-collection/maintainers-should-consider-following-node-js-release-schedule-ab08ed4de71a).
 
-**Result after synchronization:**
+## Contributing
 
-```json
-{
-    "id": "gpt-4",
-    "name": "GPT-4",
-    "cost": { "input": 0.03, "output": 0.06 },
-    "description": "GPT-4 is a large multimodal model",
-    "releaseDate": "2023-03-14"
-}
-```
+If you would like to help take a look at the [list of issues](https://github.com/anolilab/ai-models/issues) and check our [Contributing](.github/CONTRIBUTING.md) guide.
 
-## 🤝 Contributing
-
-We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.
+> **Note:** please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.
 
 ### Adding New Providers
 
@@ -562,24 +632,29 @@ We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) f
 
 Please use our [Issue Tracker](https://github.com/anolilab/ai-models/issues) to report bugs or request features.
 
-## 📄 License
+## Credits
 
-This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE.md) file for details.
+- [Daniel Bannert](https://github.com/prisis)
+- [All Contributors](https://github.com/anolilab/ai-models/graphs/contributors)
 
-## 🙏 Acknowledgments
+## Made with ❤️ at Anolilab
 
-- [Helicone](https://helicone.ai/) for providing pricing data
-- [OpenRouter](https://openrouter.ai/) for reference data
-- All the AI providers for their amazing models
-- The open source community for inspiration and tools
+This is an open source project and will always remain free to use. If you think it's cool, please star it 🌟. [Anolilab](https://www.anolilab.com/open-source) is a Development and AI Studio. Contact us at [hello@anolilab.com](mailto:hello@anolilab.com) if you need any help with these technologies or just want to say hi!
 
-## 📞 Support
+## License
 
-- **Documentation**: [GitHub Wiki](https://github.com/anolilab/ai-models/wiki)
-- **Issues**: [GitHub Issues](https://github.com/anolilab/ai-models/issues)
-- **Discussions**: [GitHub Discussions](https://github.com/anolilab/ai-models/discussions)
-- **Email**: support@anolilab.com
+The anolilab ai-model-registry is open-sourced software licensed under the [Apache License 2.0][license-url]
 
----
+<!-- badges -->
 
-Made with ❤️ by [AnoliLab](https://anolilab.com)
+[license-badge]: https://img.shields.io/badge/License-Apache--2.0-blue.svg?style=for-the-badge
+[license]: https://github.com/anolilab/ai-models/blob/main/packages/ai-model-registry/LICENSE.md
+[npm-downloads-badge]: https://img.shields.io/npm/dm/@anolilab/ai-model-registry?style=for-the-badge
+[npm-downloads]: https://www.npmjs.com/package/@anolilab/ai-model-registry
+[prs-welcome-badge]: https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=for-the-badge
+[prs-welcome]: https://github.com/anolilab/ai-models/blob/main/.github/CONTRIBUTING.md
+[chat-badge]: https://img.shields.io/discord/902465130518949899.svg?style=for-the-badge
+[chat]: https://discord.gg/TtFJY8xkFK
+[typescript-badge]: https://img.shields.io/badge/Typescript-294E80.svg?style=for-the-badge&logo=typescript
+[typescript-url]: https://www.typescriptlang.org/
+[license-url]: LICENSE.md