moai-adk 0.32.8__py3-none-any.whl → 0.41.0__py3-none-any.whl

moai_adk/templates/.claude/skills/moai-formats-data/SKILL.md

@@ -1,18 +1,21 @@
 ---
-name: moai-formats-data
-description: Data format specialist covering TOON encoding, JSON/YAML optimization, serialization patterns, and data validation for modern applications
-version: 1.0.0
-category: library
-tags:
- - formats
- - data
- - toon
- - serialization
- - validation
- - optimization
-updated: 2025-11-30
-status: active
-author: MoAI-ADK Team
+name: "moai-formats-data"
+description: "Data format specialist covering TOON encoding, JSON/YAML optimization, serialization patterns, and data validation for modern applications. Use when optimizing data for LLM transmission, implementing high-performance serialization, validating data schemas, or converting between data formats."
+version: 2.0.0
+category: "library"
+modularized: true
+tags: ['formats', 'data', 'toon', 'serialization', 'validation', 'optimization']
+updated: 2026-01-08
+allowed-tools:
+  - Read
+  - Write
+  - Edit
+  - Grep
+  - Glob
+  - mcp__context7__resolve-library-id
+  - mcp__context7__get-library-docs
+status: "active"
+author: "MoAI-ADK Team"
 ---
 
 # Data Format Specialist
@@ -21,22 +24,22 @@ author: MoAI-ADK Team
 
 Advanced Data Format Management - Comprehensive data handling covering TOON encoding, JSON/YAML optimization, serialization patterns, and data validation for performance-critical applications.
 
-Core Capabilities:
-- TOON Encoding: 40-60% token reduction vs JSON for LLM communication
-- JSON/YAML Optimization: Efficient serialization and parsing patterns
-- Data Validation: Schema validation, type checking, error handling
-- Format Conversion: Seamless transformation between data formats
-- Performance: Optimized data structures and caching strategies
-- Schema Management: Dynamic schema generation and evolution
+**Core Capabilities:**
+- **TOON Encoding**: 40-60% token reduction vs JSON for LLM communication
+- **JSON/YAML Optimization**: Efficient serialization and parsing patterns
+- **Data Validation**: Schema validation, type checking, error handling
+- **Format Conversion**: Seamless transformation between data formats
+- **Performance**: Optimized data structures and caching strategies
+- **Schema Management**: Dynamic schema generation and evolution
 
-When to Use:
+**When to Use:**
 - Optimizing data transmission to LLMs within token budgets
 - High-performance serialization/deserialization
 - Schema validation and data integrity
 - Format conversion and data transformation
 - Large dataset processing and optimization
 
-Quick Start:
+**Quick Start:**
 ```python
 # TOON encoding (40-60% token reduction)
 from moai_formats_data import TOONEncoder
@@ -62,19 +65,19 @@ result = validator.validate({"name": "John"}, schema)
 
 ### Core Concepts
 
-TOON (Token-Optimized Object Notation):
+**TOON (Token-Optimized Object Notation):**
 - Custom binary-compatible format optimized for LLM token usage
-- Type markers: `#` (numbers), `!` (booleans), `@` (timestamps), `~` (null)
+- Type markers: # (numbers), ! (booleans), @ (timestamps), ~ (null)
 - 40-60% size reduction vs JSON for typical data structures
 - Lossless round-trip encoding/decoding
 
-Performance Optimization:
+**Performance Optimization:**
 - Ultra-fast JSON processing with orjson (2-5x faster than standard json)
 - Streaming processing for large datasets using ijson
 - Intelligent caching with LRU eviction and memory management
 - Schema compression and validation optimization
 
-Data Validation:
+**Data Validation:**
 - Type-safe validation with custom rules and patterns
 - Schema evolution and migration support
 - Cross-field validation and dependency checking
@@ -89,8 +92,8 @@ from datetime import datetime
 # 1. TOON Encoding for LLM optimization
 encoder = TOONEncoder()
 data = {
- "user": {"id": 123, "name": "John", "active": True, "created": datetime.now()},
- "permissions": ["read", "write", "admin"]
+    "user": {"id": 123, "name": "John", "active": True, "created": datetime.now()},
+    "permissions": ["read", "write", "admin"]
 }
 
 # Encode and compare sizes
@@ -113,9 +116,9 @@ validator = DataValidator()
 
 # Create validation schema
 user_schema = validator.create_schema({
- "username": {"type": "string", "required": True, "min_length": 3},
- "email": {"type": "email", "required": True},
- "age": {"type": "integer", "required": False, "min_value": 13}
+    "username": {"type": "string", "required": True, "min_length": 3},
+    "email": {"type": "email", "required": True},
+    "age": {"type": "integer", "required": False, "min_value": 13}
 })
 
 # Validate data
@@ -123,28 +126,28 @@ user_data = {"username": "john_doe", "email": "john@example.com", "age": 30}
 result = validator.validate(user_data, user_schema)
 
 if result['valid']:
- print("Data is valid!")
- sanitized = result['sanitized_data']
+    print("Data is valid!")
+    sanitized = result['sanitized_data']
 else:
- print("Validation errors:", result['errors'])
+    print("Validation errors:", result['errors'])
 ```
 
 ### Common Use Cases
 
-API Response Optimization:
+**API Response Optimization:**
 ```python
 # Optimize API responses for LLM consumption
 def optimize_api_response(data: Dict) -> str:
- encoder = TOONEncoder()
- return encoder.encode(data)
+    encoder = TOONEncoder()
+    return encoder.encode(data)
 
 # Parse optimized responses
 def parse_optimized_response(toon_data: str) -> Dict:
- encoder = TOONEncoder()
- return encoder.decode(toon_data)
+    encoder = TOONEncoder()
+    return encoder.decode(toon_data)
 ```
 
-Configuration Management:
+**Configuration Management:**
 ```python
 # Fast YAML configuration loading
 from moai_formats_data import YAMLOptimizer
@@ -156,7 +159,7 @@ config = yaml_optimizer.load_fast("config.yaml")
 merged = yaml_optimizer.merge_configs(base_config, env_config, user_config)
 ```
 
-Large Dataset Processing:
+**Large Dataset Processing:**
 ```python
 # Stream processing for large JSON files
 from moai_formats_data import StreamProcessor
@@ -165,320 +168,90 @@ processor = StreamProcessor(chunk_size=8192)
 
 # Process file line by line without loading into memory
 def process_item(item):
- print(f"Processing: {item['id']}")
+    print(f"Processing: {item['id']}")
 
 processor.process_json_stream("large_dataset.json", process_item)
 ```
 
 ---
 
-## Advanced Features (10+ minutes)
+## Advanced Features Overview
 
 ### Advanced TOON Features
 
-Custom Type Handlers:
-```python
-# Extend TOON encoder with custom types
-class CustomTOONEncoder(TOONEncoder):
- def _encode_value(self, value):
- # Handle UUID objects
- if hasattr(value, 'hex') and len(value.hex) == 32: # UUID
- return f'${value.hex}'
-
- # Handle Decimal objects
- elif hasattr(value, 'as_tuple'): # Decimal
- return f'&{str(value)}'
-
- return super()._encode_value(value)
-
- def _parse_value(self, s):
- # Parse custom UUIDs
- if s.startswith('$') and len(s) == 33:
- import uuid
- return uuid.UUID(s[1:])
-
- # Parse custom Decimals
- elif s.startswith('&'):
- from decimal import Decimal
- return Decimal(s[1:])
-
- return super()._parse_value(s)
-```
-
-Streaming TOON Processing:
-```python
-# Process TOON data in streaming mode
-def stream_toon_data(data_generator):
- encoder = TOONEncoder()
- for data in data_generator:
- yield encoder.encode(data)
-
-# Batch TOON processing
-def batch_encode_toon(data_list: List[Dict], batch_size: int = 1000):
- encoder = TOONEncoder()
- results = []
-
- for i in range(0, len(data_list), batch_size):
- batch = data_list[i:i + batch_size]
- encoded_batch = [encoder.encode(item) for item in batch]
- results.extend(encoded_batch)
-
- return results
-```
+See [`modules/toon-encoding.md`](./modules/toon-encoding.md) for:
+- Custom type handlers (UUID, Decimal, etc.)
+- Streaming TOON processing
+- Batch TOON encoding
+- Performance characteristics and benchmarks
 
 ### Advanced Validation Patterns
 
-Cross-Field Validation:
-```python
-# Validate relationships between fields
-class CrossFieldValidator:
- def __init__(self):
- self.base_validator = DataValidator()
-
- def validate_user_data(self, data: Dict) -> Dict:
- # Base validation
- schema = self.base_validator.create_schema({
- "password": {"type": "string", "required": True, "min_length": 8},
- "confirm_password": {"type": "string", "required": True},
- "email": {"type": "email", "required": True}
- })
-
- result = self.base_validator.validate(data, schema)
-
- # Cross-field validation
- if data.get("password") != data.get("confirm_password"):
- result['errors']['password_mismatch'] = "Passwords do not match"
- result['valid'] = False
-
- return result
-```
-
-Schema Evolution:
-```python
-# Handle schema changes over time
-from moai_formats_data import SchemaEvolution
-
-evolution = SchemaEvolution()
-
-# Define schema versions
-v1_schema = {"name": {"type": "string"}, "age": {"type": "integer"}}
-v2_schema = {"full_name": {"type": "string"}, "age": {"type": "integer"}, "email": {"type": "email"}}
-
-# Register schemas
-evolution.register_schema("v1", v1_schema)
-evolution.register_schema("v2", v2_schema)
-
-# Add migration function
-def migrate_v1_to_v2(data: Dict) -> Dict:
- return {
- "full_name": data["name"],
- "age": data["age"],
- "email": None # New required field
- }
-
-evolution.add_migration("v1", "v2", migrate_v1_to_v2)
-
-# Migrate data
-old_data = {"name": "John Doe", "age": 30}
-new_data = evolution.migrate_data(old_data, "v1", "v2")
-```
+See [`modules/data-validation.md`](./modules/data-validation.md) for:
+- Cross-field validation
+- Schema evolution and migration
+- Custom validation rules
+- Batch validation optimization
 
 ### Performance Optimization
 
-Intelligent Caching:
-```python
-from moai_formats_data import SmartCache
-
-# Create cache with memory constraints
-cache = SmartCache(max_memory_mb=50, max_items=10000)
-
-@cache.cache.cache_result(ttl=1800) # 30 minutes
-def expensive_data_processing(data: Dict) -> Dict:
- # Simulate expensive computation
- time.sleep(0.1)
- return {"processed": True, "data": data}
-
-# Cache statistics
-print(cache.get_stats())
+See [`modules/caching-performance.md`](./modules/caching-performance.md) for:
+- Intelligent caching strategies
+- Cache warming and invalidation
+- Memory management
+- Performance monitoring
 
-# Cache warming
-def warm_common_data():
- common_queries = [
- {"type": "user", "id": 1},
- {"type": "user", "id": 2},
- {"type": "config", "key": "app"}
- ]
+### JSON/YAML Advanced Features
 
- for query in common_queries:
- expensive_data_processing(query)
-
-warm_common_data()
-```
-
-Batch Processing Optimization:
-```python
-# Optimized batch validation
-def validate_batch_optimized(data_list: List[Dict], schema: Dict) -> List[Dict]:
- validator = DataValidator()
-
- # Pre-compile patterns for performance
- validator._compile_schema_patterns(schema)
-
- # Process in batches for memory efficiency
- batch_size = 1000
- results = []
-
- for i in range(0, len(data_list), batch_size):
- batch = data_list[i:i + batch_size]
- batch_results = [validator.validate(data, schema) for data in batch]
- results.extend(batch_results)
-
- return results
-```
-
-### Integration Patterns
-
-LLM Integration:
-```python
-# Prepare data for LLM consumption
-def prepare_for_llm(data: Dict, max_tokens: int = 2000) -> str:
- encoder = TOONEncoder()
- toon_data = encoder.encode(data)
-
- # Check token count
- estimated_tokens = len(toon_data.split())
-
- if estimated_tokens > max_tokens:
- # Implement data reduction strategy
- reduced_data = reduce_data_complexity(data, max_tokens)
- toon_data = encoder.encode(reduced_data)
-
- return toon_data
-
-def reduce_data_complexity(data: Dict, max_tokens: int) -> Dict:
- """Reduce data complexity to fit token budget."""
- # Implement selective field removal
- priority_fields = ["id", "name", "email", "status"]
- reduced = {k: v for k, v in data.items() if k in priority_fields}
-
- # Further reduction if needed
- encoder = TOONEncoder()
- while len(encoder.encode(reduced).split()) > max_tokens:
- # Remove least important fields
- if len(reduced) <= 1:
- break
- reduced.popitem()
-
- return reduced
-```
-
-Database Integration:
-```python
-# Optimize database queries with format conversion
-def optimize_db_response(db_data: List[Dict]) -> Dict:
- # Convert database results to optimized format
- optimizer = JSONOptimizer()
-
- # Compress and cache schema
- common_schema = {"type": "object", "properties": {"id": {"type": "integer"}}}
- compressed_schema = optimizer.compress_schema(common_schema)
-
- # Process in batches
- processor = StreamProcessor()
- processed_data = []
-
- for item in db_data:
- # Apply validation and transformation
- processed_item = transform_db_item(item)
- processed_data.append(processed_item)
-
- return {
- "data": processed_data,
- "count": len(processed_data),
- "schema": compressed_schema
- }
-```
+See [`modules/json-optimization.md`](./modules/json-optimization.md) for:
+- Streaming JSON processing
+- Memory-efficient parsing
+- Schema compression
+- Format conversion utilities
 
 ---
 
 ## Works Well With
 
-- moai-domain-backend - Backend data serialization and API responses
-- moai-domain-database - Database data format optimization
-- moai-integration-mcp - MCP data serialization and transmission
-- moai-docs-generation - Documentation data formatting
-- moai-foundation-core - Core data architecture principles
+- **moai-domain-backend** - Backend data serialization and API responses
+- **moai-domain-database** - Database data format optimization
+- **moai-foundation-core** - MCP data serialization and transmission patterns
+- **moai-workflow-docs** - Documentation data formatting
+- **moai-foundation-context** - Context optimization for token budgets
 
 ---
 
 ## Module References
 
-Core Implementation Modules:
-- [`modules/toon-encoding.md`](./modules/toon-encoding.md) - TOON encoding implementation and examples
-- [`modules/json-optimization.md`](./modules/json-optimization.md) - High-performance JSON/YAML processing
-- [`modules/data-validation.md`](./modules/data-validation.md) - Advanced validation and schema management
-- [`modules/caching-performance.md`](./modules/caching-performance.md) - Caching strategies and performance optimization
-
----
-
-## Usage Examples
-
-### CLI Usage
-```bash
-# Encode data to TOON format
-moai-formats encode-toon --input data.json --output data.toon
-
-# Validate data against schema
-moai-formats validate --schema schema.json --data data.json
-
-# Convert between formats
-moai-formats convert --input data.json --output data.yaml --format yaml
-
-# Optimize JSON structure
-moai-formats optimize-json --input large-data.json --output optimized.json
-```
-
-### Python API
-```python
-from moai_formats_data import TOONEncoder, DataValidator, JSONOptimizer
-
-# TOON encoding
-encoder = TOONEncoder()
-toon_data = encoder.encode({"user": "John", "age": 30})
-original_data = encoder.decode(toon_data)
+**Core Implementation Modules:**
+- [`modules/toon-encoding.md`](./modules/toon-encoding.md) - TOON encoding implementation (308 lines)
+- [`modules/json-optimization.md`](./modules/json-optimization.md) - High-performance JSON/YAML (374 lines)
+- [`modules/data-validation.md`](./modules/data-validation.md) - Advanced validation and schemas (485 lines)
+- [`modules/caching-performance.md`](./modules/caching-performance.md) - Caching strategies (459 lines)
 
-# Data validation
-validator = DataValidator()
-schema = validator.create_schema({
- "name": {"type": "string", "required": True, "min_length": 2},
- "email": {"type": "email", "required": True}
-})
-result = validator.validate({"name": "John", "email": "john@example.com"}, schema)
-
-# JSON optimization
-optimizer = JSONOptimizer()
-fast_json = optimizer.serialize_fast(large_dataset)
-parsed_data = optimizer.deserialize_fast(fast_json)
-```
+**Supporting Files:**
+- [`modules/README.md`](./modules/README.md) - Module overview and integration patterns (98 lines)
+- [`reference.md`](./reference.md) - Extended reference documentation (585 lines)
+- [`examples.md`](./examples.md) - Complete working examples (804 lines)
 
 ---
 
 ## Technology Stack
 
-Core Libraries:
+**Core Libraries:**
 - orjson: Ultra-fast JSON parsing and serialization
 - PyYAML: YAML processing with C-based loaders
 - ijson: Streaming JSON parser for large files
 - python-dateutil: Advanced datetime parsing
 - regex: Advanced regular expression support
 
-Performance Tools:
+**Performance Tools:**
 - lru_cache: Built-in memoization
 - pickle: Object serialization
 - hashlib: Hash generation for caching
 - functools: Function decorators and utilities
 
-Validation Libraries:
+**Validation Libraries:**
 - jsonschema: JSON Schema validation
 - cerberus: Lightweight data validation
 - marshmallow: Object serialization/deserialization
@@ -486,6 +259,22 @@ Validation Libraries:
 
 ---
 
-Status: Production Ready
-Last Updated: 2025-11-30
-Maintained by: MoAI-ADK Data Team
+## Version History
+
+**v2.0.0 (2026-01-06):**
+- Modularized skill with progressive disclosure pattern
+- Reduced SKILL.md from 493 to 247 lines (50% reduction)
+- Organized advanced content into focused modules
+- Enhanced module cross-references and integration patterns
+- Updated Context7 MCP tools in allowed-tools
+
+**v1.0.0 (2025-12-06):**
+- Initial release with monolithic structure
+- TOON encoding, JSON/YAML optimization, data validation
+- Basic caching and performance features
+
+---
+
+**Status:** Production Ready
+**Last Updated:** 2026-01-06
+**Maintained by:** MoAI-ADK Data Team