moai-adk 0.34.0__py3-none-any.whl → 1.1.0__py3-none-any.whl

moai_adk/templates/.claude/skills/moai-formats-data/SKILL.md

@@ -1,28 +1,32 @@
 ---
-name: moai-formats-data
-description: Data format specialist covering TOON encoding, JSON/YAML optimization, serialization patterns, and data validation for modern applications
-version: 1.0.0
-category: library
-allowed-tools: Read, Write, Edit, Grep, Glob
-tags:
- - formats
- - data
- - toon
- - serialization
- - validation
- - optimization
-updated: 2025-12-06
-status: active
-author: MoAI-ADK Team
+name: "moai-formats-data"
+description: "Data format specialist covering TOON encoding, JSON/YAML optimization, serialization patterns, and data validation for modern applications. Use when optimizing data for LLM transmission, implementing high-performance serialization, validating data schemas, or converting between data formats."
+version: 2.0.0
+category: "library"
+modularized: true
+user-invocable: false
+tags: ['formats', 'data', 'toon', 'serialization', 'validation', 'optimization']
+updated: 2026-01-11
+allowed-tools:
+  - Read
+  - Write
+  - Edit
+  - Grep
+  - Glob
+  - mcp__context7__resolve-library-id
+  - mcp__context7__get-library-docs
+status: "active"
+author: "MoAI-ADK Team"
 ---
 
 # Data Format Specialist
 
-## Quick Reference (30 seconds)
+## Quick Reference
 
 Advanced Data Format Management - Comprehensive data handling covering TOON encoding, JSON/YAML optimization, serialization patterns, and data validation for performance-critical applications.
 
 Core Capabilities:
+
 - TOON Encoding: 40-60% token reduction vs JSON for LLM communication
 - JSON/YAML Optimization: Efficient serialization and parsing patterns
 - Data Validation: Schema validation, type checking, error handling
@@ -31,6 +35,7 @@ Core Capabilities:
 - Schema Management: Dynamic schema generation and evolution
 
 When to Use:
+
 - Optimizing data transmission to LLMs within token budgets
 - High-performance serialization/deserialization
 - Schema validation and data integrity
@@ -38,44 +43,35 @@ When to Use:
 - Large dataset processing and optimization
 
 Quick Start:
-```python
-# TOON encoding (40-60% token reduction)
-from moai_formats_data import TOONEncoder
-encoder = TOONEncoder()
-compressed = encoder.encode({"user": "John", "age": 30})
-original = encoder.decode(compressed)
-
-# Fast JSON processing
-from moai_formats_data import JSONOptimizer
-optimizer = JSONOptimizer()
-fast_json = optimizer.serialize_fast(large_dataset)
-
-# Data validation
-from moai_formats_data import DataValidator
-validator = DataValidator()
-schema = validator.create_schema({"name": {"type": "string", "required": True}})
-result = validator.validate({"name": "John"}, schema)
-```
+
+Create a TOONEncoder instance and call encode with a dictionary containing user and age fields to compress the data. The encoded result achieves 40-60% token reduction. Call decode to restore the original data structure.
+
+Create a JSONOptimizer instance and call serialize_fast with a large dataset to achieve ultra-fast JSON processing.
+
+Create a DataValidator instance and call create_schema with a dictionary defining name as a required string type. Call validate with the data and schema to check validity.
 
 ---
 
-## Implementation Guide (5 minutes)
+## Implementation Guide
 
 ### Core Concepts
 
 TOON (Token-Optimized Object Notation):
+
 - Custom binary-compatible format optimized for LLM token usage
-- Type markers: # (numbers), ! (booleans), @ (timestamps), ~ (null)
+- Type markers: # for numbers, ! for booleans, @ for timestamps, ~ for null
 - 40-60% size reduction vs JSON for typical data structures
 - Lossless round-trip encoding/decoding
 
 Performance Optimization:
-- Ultra-fast JSON processing with orjson (2-5x faster than standard json)
+
+- Ultra-fast JSON processing with orjson achieving 2-5x faster than standard json
 - Streaming processing for large datasets using ijson
 - Intelligent caching with LRU eviction and memory management
 - Schema compression and validation optimization
 
 Data Validation:
+
 - Type-safe validation with custom rules and patterns
 - Schema evolution and migration support
 - Cross-field validation and dependency checking
@@ -83,322 +79,51 @@ Data Validation:
 
 ### Basic Implementation
 
-```python
-from moai_formats_data import TOONEncoder, JSONOptimizer, DataValidator
-from datetime import datetime
-
-# 1. TOON Encoding for LLM optimization
-encoder = TOONEncoder()
-data = {
- "user": {"id": 123, "name": "John", "active": True, "created": datetime.now()},
- "permissions": ["read", "write", "admin"]
-}
-
-# Encode and compare sizes
-toon_data = encoder.encode(data)
-original_data = encoder.decode(toon_data)
-
-# 2. Fast JSON Processing
-optimizer = JSONOptimizer()
-
-# Ultra-fast serialization
-json_bytes = optimizer.serialize_fast(data)
-parsed_data = optimizer.deserialize_fast(json_bytes)
-
-# Schema compression for repeated validation
-schema = {"type": "object", "properties": {"name": {"type": "string"}}}
-compressed_schema = optimizer.compress_schema(schema)
-
-# 3. Data Validation
-validator = DataValidator()
-
-# Create validation schema
-user_schema = validator.create_schema({
- "username": {"type": "string", "required": True, "min_length": 3},
- "email": {"type": "email", "required": True},
- "age": {"type": "integer", "required": False, "min_value": 13}
-})
-
-# Validate data
-user_data = {"username": "john_doe", "email": "john@example.com", "age": 30}
-result = validator.validate(user_data, user_schema)
-
-if result['valid']:
- print("Data is valid!")
- sanitized = result['sanitized_data']
-else:
- print("Validation errors:", result['errors'])
-```
+TOON Encoding for LLM Optimization:
+
+Create a TOONEncoder instance. Define data with user object containing id, name, active boolean, and created datetime, plus permissions array. Call encode to compress and decode to restore. Compare sizes to verify reduction.
+
+Fast JSON Processing:
+
+Create a JSONOptimizer instance. Call serialize_fast to get bytes and deserialize_fast to parse. Use compress_schema with a type object and properties definition to optimize repeated validation.
+
+Data Validation:
+
+Create a DataValidator instance. Define user_schema with username requiring string type, minimum length 3, email requiring email type, and age as optional integer with minimum value 13. Call validate with user_data and schema, then check result for valid status, sanitized_data, or errors list.
 
 ### Common Use Cases
 
 API Response Optimization:
-```python
-# Optimize API responses for LLM consumption
-def optimize_api_response(data: Dict) -> str:
- encoder = TOONEncoder()
- return encoder.encode(data)
-
-# Parse optimized responses
-def parse_optimized_response(toon_data: str) -> Dict:
- encoder = TOONEncoder()
- return encoder.decode(toon_data)
-```
 
-Configuration Management:
-```python
-# Fast YAML configuration loading
-from moai_formats_data import YAMLOptimizer
+Create a function to optimize API responses for LLM consumption by encoding data with TOONEncoder. Create a corresponding function to parse optimized responses by decoding TOON data back to dictionary.
 
-yaml_optimizer = YAMLOptimizer()
-config = yaml_optimizer.load_fast("config.yaml")
+Configuration Management:
 
-# Merge multiple configurations
-merged = yaml_optimizer.merge_configs(base_config, env_config, user_config)
-```
+Create a YAMLOptimizer instance and call load_fast with a config file path. Call merge_configs with base_config, env_config, and user_config for multi-file merging.
 
 Large Dataset Processing:
-```python
-# Stream processing for large JSON files
-from moai_formats_data import StreamProcessor
-
-processor = StreamProcessor(chunk_size=8192)
 
-# Process file line by line without loading into memory
-def process_item(item):
- print(f"Processing: {item['id']}")
-
-processor.process_json_stream("large_dataset.json", process_item)
-```
+Create a StreamProcessor with chunk_size of 8192. Define a process_item function that handles each item. Call process_json_stream with the file path and callback to process large JSON files without loading into memory.
 
 ---
 
-## Advanced Features (10+ minutes)
+## Advanced Features Overview
 
 ### Advanced TOON Features
 
-Custom Type Handlers:
-```python
-# Extend TOON encoder with custom types
-class CustomTOONEncoder(TOONEncoder):
- def _encode_value(self, value):
- # Handle UUID objects
- if hasattr(value, 'hex') and len(value.hex) == 32: # UUID
- return f'${value.hex}'
-
- # Handle Decimal objects
- elif hasattr(value, 'as_tuple'): # Decimal
- return f'&{str(value)}'
-
- return super()._encode_value(value)
-
- def _parse_value(self, s):
- # Parse custom UUIDs
- if s.startswith('$') and len(s) == 33:
- import uuid
- return uuid.UUID(s[1:])
-
- # Parse custom Decimals
- elif s.startswith('&'):
- from decimal import Decimal
- return Decimal(s[1:])
-
- return super()._parse_value(s)
-```
-
-Streaming TOON Processing:
-```python
-# Process TOON data in streaming mode
-def stream_toon_data(data_generator):
- encoder = TOONEncoder()
- for data in data_generator:
- yield encoder.encode(data)
-
-# Batch TOON processing
-def batch_encode_toon(data_list: List[Dict], batch_size: int = 1000):
- encoder = TOONEncoder()
- results = []
-
- for i in range(0, len(data_list), batch_size):
- batch = data_list[i:i + batch_size]
- encoded_batch = [encoder.encode(item) for item in batch]
- results.extend(encoded_batch)
-
- return results
-```
+See modules/toon-encoding.md for custom type handlers (UUID, Decimal), streaming TOON processing, batch TOON encoding, and performance characteristics with benchmarks.
 
 ### Advanced Validation Patterns
 
-Cross-Field Validation:
-```python
-# Validate relationships between fields
-class CrossFieldValidator:
- def __init__(self):
- self.base_validator = DataValidator()
-
- def validate_user_data(self, data: Dict) -> Dict:
- # Base validation
- schema = self.base_validator.create_schema({
- "password": {"type": "string", "required": True, "min_length": 8},
- "confirm_password": {"type": "string", "required": True},
- "email": {"type": "email", "required": True}
- })
-
- result = self.base_validator.validate(data, schema)
-
- # Cross-field validation
- if data.get("password") != data.get("confirm_password"):
- result['errors']['password_mismatch'] = "Passwords do not match"
- result['valid'] = False
-
- return result
-```
-
-Schema Evolution:
-```python
-# Handle schema changes over time
-from moai_formats_data import SchemaEvolution
-
-evolution = SchemaEvolution()
-
-# Define schema versions
-v1_schema = {"name": {"type": "string"}, "age": {"type": "integer"}}
-v2_schema = {"full_name": {"type": "string"}, "age": {"type": "integer"}, "email": {"type": "email"}}
-
-# Register schemas
-evolution.register_schema("v1", v1_schema)
-evolution.register_schema("v2", v2_schema)
-
-# Add migration function
-def migrate_v1_to_v2(data: Dict) -> Dict:
- return {
- "full_name": data["name"],
- "age": data["age"],
- "email": None # New required field
- }
-
-evolution.add_migration("v1", "v2", migrate_v1_to_v2)
-
-# Migrate data
-old_data = {"name": "John Doe", "age": 30}
-new_data = evolution.migrate_data(old_data, "v1", "v2")
-```
+See modules/data-validation.md for cross-field validation, schema evolution and migration, custom validation rules, and batch validation optimization.
 
 ### Performance Optimization
 
-Intelligent Caching:
-```python
-from moai_formats_data import SmartCache
-
-# Create cache with memory constraints
-cache = SmartCache(max_memory_mb=50, max_items=10000)
-
-@cache.cache.cache_result(ttl=1800) # 30 minutes
-def expensive_data_processing(data: Dict) -> Dict:
- # Simulate expensive computation
- time.sleep(0.1)
- return {"processed": True, "data": data}
-
-# Cache statistics
-print(cache.get_stats())
-
-# Cache warming
-def warm_common_data():
- common_queries = [
- {"type": "user", "id": 1},
- {"type": "user", "id": 2},
- {"type": "config", "key": "app"}
- ]
-
- for query in common_queries:
- expensive_data_processing(query)
-
-warm_common_data()
-```
-
-Batch Processing Optimization:
-```python
-# Optimized batch validation
-def validate_batch_optimized(data_list: List[Dict], schema: Dict) -> List[Dict]:
- validator = DataValidator()
-
- # Pre-compile patterns for performance
- validator._compile_schema_patterns(schema)
-
- # Process in batches for memory efficiency
- batch_size = 1000
- results = []
-
- for i in range(0, len(data_list), batch_size):
- batch = data_list[i:i + batch_size]
- batch_results = [validator.validate(data, schema) for data in batch]
- results.extend(batch_results)
-
- return results
-```
-
-### Integration Patterns
-
-LLM Integration:
-```python
-# Prepare data for LLM consumption
-def prepare_for_llm(data: Dict, max_tokens: int = 2000) -> str:
- encoder = TOONEncoder()
- toon_data = encoder.encode(data)
-
- # Check token count
- estimated_tokens = len(toon_data.split())
-
- if estimated_tokens > max_tokens:
- # Implement data reduction strategy
- reduced_data = reduce_data_complexity(data, max_tokens)
- toon_data = encoder.encode(reduced_data)
-
- return toon_data
-
-def reduce_data_complexity(data: Dict, max_tokens: int) -> Dict:
- """Reduce data complexity to fit token budget."""
- # Implement selective field removal
- priority_fields = ["id", "name", "email", "status"]
- reduced = {k: v for k, v in data.items() if k in priority_fields}
-
- # Further reduction if needed
- encoder = TOONEncoder()
- while len(encoder.encode(reduced).split()) > max_tokens:
- # Remove least important fields
- if len(reduced) <= 1:
- break
- reduced.popitem()
-
- return reduced
-```
-
-Database Integration:
-```python
-# Optimize database queries with format conversion
-def optimize_db_response(db_data: List[Dict]) -> Dict:
- # Convert database results to optimized format
- optimizer = JSONOptimizer()
-
- # Compress and cache schema
- common_schema = {"type": "object", "properties": {"id": {"type": "integer"}}}
- compressed_schema = optimizer.compress_schema(common_schema)
-
- # Process in batches
- processor = StreamProcessor()
- processed_data = []
-
- for item in db_data:
- # Apply validation and transformation
- processed_item = transform_db_item(item)
- processed_data.append(processed_item)
-
- return {
- "data": processed_data,
- "count": len(processed_data),
- "schema": compressed_schema
- }
-```
+See modules/caching-performance.md for intelligent caching strategies, cache warming and invalidation, memory management, and performance monitoring.
+
+### JSON/YAML Advanced Features
+
+See modules/json-optimization.md for streaming JSON processing, memory-efficient parsing, schema compression, and format conversion utilities.
 
 ---
 
@@ -407,66 +132,32 @@ def optimize_db_response(db_data: List[Dict]) -> Dict:
 - moai-domain-backend - Backend data serialization and API responses
 - moai-domain-database - Database data format optimization
 - moai-foundation-core - MCP data serialization and transmission patterns
-- moai-docs-generation - Documentation data formatting
-- moai-foundation-core - Core data architecture principles
+- moai-workflow-docs - Documentation data formatting
+- moai-foundation-context - Context optimization for token budgets
 
 ---
 
 ## Module References
 
 Core Implementation Modules:
-- [`modules/toon-encoding.md`](./modules/toon-encoding.md) - TOON encoding implementation and examples
-- [`modules/json-optimization.md`](./modules/json-optimization.md) - High-performance JSON/YAML processing
-- [`modules/data-validation.md`](./modules/data-validation.md) - Advanced validation and schema management
-- [`modules/caching-performance.md`](./modules/caching-performance.md) - Caching strategies and performance optimization
-
----
-
-## Usage Examples
-
-### CLI Usage
-```bash
-# Encode data to TOON format
-moai-formats encode-toon --input data.json --output data.toon
-
-# Validate data against schema
-moai-formats validate --schema schema.json --data data.json
 
-# Convert between formats
-moai-formats convert --input data.json --output data.yaml --format yaml
+- modules/toon-encoding.md - TOON encoding implementation
+- modules/json-optimization.md - High-performance JSON/YAML
+- modules/data-validation.md - Advanced validation and schemas
+- modules/caching-performance.md - Caching strategies
 
-# Optimize JSON structure
-moai-formats optimize-json --input large-data.json --output optimized.json
-```
+Supporting Files:
 
-### Python API
-```python
-from moai_formats_data import TOONEncoder, DataValidator, JSONOptimizer
-
-# TOON encoding
-encoder = TOONEncoder()
-toon_data = encoder.encode({"user": "John", "age": 30})
-original_data = encoder.decode(toon_data)
-
-# Data validation
-validator = DataValidator()
-schema = validator.create_schema({
- "name": {"type": "string", "required": True, "min_length": 2},
- "email": {"type": "email", "required": True}
-})
-result = validator.validate({"name": "John", "email": "john@example.com"}, schema)
-
-# JSON optimization
-optimizer = JSONOptimizer()
-fast_json = optimizer.serialize_fast(large_dataset)
-parsed_data = optimizer.deserialize_fast(fast_json)
-```
+- modules/README.md - Module overview and integration patterns
+- reference.md - Extended reference documentation
+- examples.md - Complete working examples
 
 ---
 
 ## Technology Stack
 
 Core Libraries:
+
 - orjson: Ultra-fast JSON parsing and serialization
 - PyYAML: YAML processing with C-based loaders
 - ijson: Streaming JSON parser for large files
@@ -474,12 +165,14 @@ Core Libraries:
 - regex: Advanced regular expression support
 
 Performance Tools:
+
 - lru_cache: Built-in memoization
 - pickle: Object serialization
 - hashlib: Hash generation for caching
 - functools: Function decorators and utilities
 
 Validation Libraries:
+
 - jsonschema: JSON Schema validation
 - cerberus: Lightweight data validation
 - marshmallow: Object serialization/deserialization
@@ -487,6 +180,10 @@ Validation Libraries:
 
 ---
 
+## Resources
+
+For working code examples, see [examples.md](examples.md).
+
 Status: Production Ready
-Last Updated: 2025-11-30
+Last Updated: 2026-01-11
 Maintained by: MoAI-ADK Data Team