cnhkmcp 2.2.0__py3-none-any.whl → 2.3.0__py3-none-any.whl

cnhkmcp/untracked/APP/trailSomeAlphas/skills/brain-data-feature-engineering/SKILL.md

@@ -0,0 +1,503 @@
+---
+brain-data-feature-engineering methodology
+---
+
+# BRAIN Data Feature Engineering Workflow
+
+**Purpose**: Automatically transform BRAIN dataset fields into deep, meaningful feature engineering ideas.
+
+## Input Requirements
+
+### Required Parameters:
+- **data_category**: Dataset category (e.g., "fundamental", "analyst", "news", "model")
+- **delay**: Data delay setting (0 or 1)
+- **region**: Market region (e.g., "USA", "EUR", "ASI")
+
+### Optional Parameters:
+- **universe**: Trading universe (default: "TOP3000")
+- **dataset_id**: Specific dataset ID (if known, skips discovery phase)
+
+## Workflow Overview
+
+
+### Step 2: Field Extraction and Deconstruction
+- **Deconstruct each field's meaning**:
+  * What is being measured? (the entity/concept)
+  * How is it measured? (collection/calculation method)
+  * Time dimension? (instantaneous, cumulative, rate of change)
+  * Business context? (why does this field exist?)
+  * Generation logic? (reliability considerations)
+- **Build field profiles**: Structured understanding of each field's essence
+
+### Step 3: Reasoning and Analysis
+**performs deep analysis based on collected information:**
+
+**A. Field Relationship Mapping**
+- Analyze logical connections between fields
+- Identify: independent fields, related fields, complementary fields
+- Map the "story" the dataset tells
+- **Key question**: What relationships are implied by these fields?
+
+**B. Question-Driven Feature Generation (Internal Process)**
+The skill asks itself these questions and generates feature concepts:
+
+1. **"What is stable?"** → Look for invariants
+   - Which fields or combinations remain relatively constant?
+   - What stability measures make sense?
+
+2. **"What is changing?"** → Analyze change patterns
+   - Rate of change, acceleration, volatility
+   - Trend vs. noise separation
+
+3. **"What is anomalous?"** → Identify deviations
+   - Outliers, unusual patterns, breaks from normal
+   - Deviation magnitude and significance
+
+4. **"What is combined?"** → Examine interactions
+   - How fields interact, amplify, or offset each other
+   - Synthesis creates new meaning
+
+5. **"What is structural?"** → Study compositions
+   - Constituent parts, proportional relationships
+   - Structural changes over time
+
+6. **"What is cumulative?"** → Explore accumulation effects
+   - Building up over time, decay effects
+   - Memory and persistence in data
+
+7. **"What is relative?"** → Make comparisons
+   - Relative positioning, ranking, normalization
+   - Context within dataset
+
+8. **"What is essential?"** → Distill to core meaning
+   - First principles thinking
+   - Strip away assumptions, get to essence
+
+**C. Feature Concept Generation**
+For each relevant question-field combination:
+- Formulate feature concept that answers the question
+- Define the concept clearly
+- Identify the logical meaning
+- Consider directionality (what high/low values mean)
+- Identify boundary conditions
+- Note potential issues/limitations
+
+### Step 4: Feature Documentation
+**For each generated feature concept, document:**
+- **Concept Name**: Clear, descriptive name
+- **Definition**: One-sentence definition
+- **Logical Meaning**: What phenomenon/concept does it represent?
+- **Why It's Meaningful**: Why does this feature make sense?
+- **Directionality**: Interpretation of high vs. low values
+- **Boundary Conditions**: What extremes indicate
+- **Data Requirements**: What fields are used and any constraints
+- **Potential Issues**: Known limitations or concerns
+
+### Step 5: Output Generation
+**Generate structured markdown report including:**
+
+0. **Output the report markdown format** in the following format:
+
+    # {dataset_name} Feature Engineering Analysis Report
+
+    **Dataset**: {dataset_id}
+    **Category**: {category}
+    **Region**: {region}
+    **Analysis Date**: {analysis_date}
+    **Fields Analyzed**: {field_count}
+
+    ---
+
+    ## Executive Summary
+
+    **Primary Question Answered by Dataset**: What does this dataset fundamentally measure?
+
+    **Key Insights from Analysis**:
+    - {insight_1}
+    - {insight_2}
+    - {insight_3}
+
+    **Critical Field Relationships Identified**:
+    - {relationship_1}
+    - {relationship_2}
+
+    **Most Promising Feature Concepts**:
+    1. {top_feature_1} - because {reason_1}
+    2. {top_feature_2} - because {reason_2}
+    3. {top_feature_3} - because {reason_3}
+
+    ---
+
+    ## Dataset Deep Understanding
+
+    ### Dataset Description
+    {dataset_description}
+
+    ### Field Inventory
+    | Field ID | Description | Data Type | Update Frequency | Coverage |
+    |----------|-------------|-----------|------------------|----------|
+    | {field_1_id} | {field_1_desc} | {type_1} | {freq_1} | {coverage_1}% |
+    | {field_2_id} | {field_2_desc} | {type_2} | {freq_2} | {coverage_2}% |
+    | {field_3_id} | {field_3_desc} | {type_3} | {freq_3} | {coverage_3}% |
+
+    *(Additional fields as needed)*
+
+    ### Field Deconstruction Analysis
+
+    #### {field_1_id}: {field_1_name}
+    - **What is being measured?**: {measurement_object_1}
+    - **How is it measured?**: {measurement_method_1}
+    - **Time dimension**: {time_dimension_1}
+    - **Business context**: {business_context_1}
+    - **Generation logic**: {generation_logic_1}
+    - **Reliability considerations**: {reliability_1}
+
+    #### {field_2_id}: {field_2_name}
+    - **What is being measured?**: {measurement_object_2}
+    - **How is it measured?**: {measurement_method_2}
+    - **Time dimension**: {time_dimension_2}
+    - **Business context**: {business_context_2}
+    - **Generation logic**: {generation_logic_2}
+    - **Reliability considerations**: {reliability_2}
+
+    *(Additional fields as needed)*
+
+    ### Field Relationship Mapping
+
+    **The Story This Data Tells**:
+    {story_description}
+
+    **Key Relationships Identified**:
+    1. {relationship_1_desc}
+    2. {relationship_2_desc}
+    3. {relationship_3_desc}
+
+    **Missing Pieces That Would Complete the Picture**:
+    - {missing_1}
+    - {missing_2}
+
+    ---
+
+    ## Feature Concepts by Question Type
+
+    ### Q1: "What is stable?" (Invariance Features)
+
+    **Concept**: {stability_feature_1_name}
+    - **Sample Fields Used**: fields_used_1
+    - **Definition**: {definition_1}
+    - **Why This Feature**: {why_1}
+    - **Logical Meaning**: {logical_meaning_1}
+    - **Directionality**: {directionality_1}
+    - **Boundary Conditions**: {boundaries_1}
+    - **Implementation Example**: `{implementation_1}`
+
+    **Concept**: {stability_feature_2_name}
+    - **Sample Fields Used**: fields_used_2
+    - **Definition**: {definition_2}
+    - **Why This Feature**: {why_2}
+    - **Logical Meaning**: {logical_meaning_2}
+    - **Directionality**: {directionality_2}
+    - **Boundary Conditions**: {boundaries_2}
+    - **Implementation Example**: `{implementation_2}`
+
+    ---
+
+    ### Q2: "What is changing?" (Dynamics Features)
+
+    **Concept**: {dynamics_feature_1_name}
+    - **Sample Fields Used**: fields_used_3
+    - **Definition**: {definition_3}
+    - **Why This Feature**: {why_3}
+    - **Logical Meaning**: {logical_meaning_3}
+    - **Directionality**: {directionality_3}
+    - **Boundary Conditions**: {boundaries_3}
+    - **Implementation Example**: `{implementation_3}`
+
+    **Concept**: {dynamics_feature_2_name}
+    - **Sample Fields Used**: fields_used_4
+    - **Definition**: {definition_4}
+    - **Why This Feature**: {why_4}
+    - **Logical Meaning**: {logical_meaning_4}
+    - **Directionality**: {directionality_4}
+    - **Boundary Conditions**: {boundaries_4}
+    - **Implementation Example**: `{implementation_4}`
+
+    ---
+
+    ### Q3: "What is anomalous?" (Deviation Features)
+
+    **Concept**: {anomaly_feature_1_name}
+    - **Sample Fields Used**: fields_used_5
+    - **Definition**: {definition_5}
+    - **Why This Feature**: {why_5}
+    - **Logical Meaning**: {logical_meaning_5}
+    - **Directionality**: {directionality_5}
+    - **Boundary Conditions**: {boundaries_5}
+    - **Implementation Example**: `{implementation_5}`
+
+    **Concept**: {anomaly_feature_2_name}
+    - **Sample Fields Used**: fields_used_6
+    - **Definition**: {definition_6}
+    - **Why This Feature**: {why_6}
+    - **Logical Meaning**: {logical_meaning_6}
+    - **Directionality**: {directionality_6}
+    - **Boundary Conditions**: {boundaries_6}
+    - **Implementation Example**: `{implementation_6}`
+
+    ---
+
+    ### Q4: "What is combined?" (Interaction Features)
+
+    **Concept**: {interaction_feature_1_name}
+    - **Sample Fields Used**: fields_used_7
+    - **Definition**: {definition_7}
+    - **Why This Feature**: {why_7}
+    - **Logical Meaning**: {logical_meaning_7}
+    - **Directionality**: {directionality_7}
+    - **Boundary Conditions**: {boundaries_7}
+    - **Implementation Example**: `{implementation_7}`
+
+    **Concept**: {interaction_feature_2_name}
+    - **Sample Fields Used**: fields_used_8
+    - **Definition**: {definition_8}
+    - **Why This Feature**: {why_8}
+    - **Logical Meaning**: {logical_meaning_8}
+    - **Directionality**: {directionality_8}
+    - **Boundary Conditions**: {boundaries_8}
+    - **Implementation Example**: `{implementation_8}`
+
+    ---
+
+    ### Q5: "What is structural?" (Composition Features)
+
+    **Concept**: {structure_feature_1_name}
+    - **Sample Fields Used**: fields_used_9
+    - **Definition**: {definition_9}
+    - **Why This Feature**: {why_9}
+    - **Logical Meaning**: {logical_meaning_9}
+    - **Directionality**: {directionality_9}
+    - **Boundary Conditions**: {boundaries_9}
+    - **Implementation Example**: `{implementation_9}`
+
+    **Concept**: {structure_feature_2_name}
+    - **Sample Fields Used**: fields_used_10
+    - **Definition**: {definition_10}
+    - **Why This Feature**: {why_10}
+    - **Logical Meaning**: {logical_meaning_10}
+    - **Directionality**: {directionality_10}
+    - **Boundary Conditions**: {boundaries_10}
+    - **Implementation Example**: `{implementation_10}`
+
+    ---
+
+    ### Q6: "What is cumulative?" (Accumulation Features)
+
+    **Concept**: {accumulation_feature_1_name}
+    - **Sample Fields Used**: fields_used_11
+    - **Definition**: {definition_11}
+    - **Why This Feature**: {why_11}
+    - **Logical Meaning**: {logical_meaning_11}
+    - **Directionality**: {directionality_11}
+    - **Boundary Conditions**: {boundaries_11}
+    - **Implementation Example**: `{implementation_11}`
+
+    **Concept**: {accumulation_feature_2_name}
+    - **Sample Fields Used**: fields_used_12
+    - **Definition**: {definition_12}
+    - **Why This Feature**: {why_12}
+    - **Logical Meaning**: {logical_meaning_12}
+    - **Directionality**: {directionality_12}
+    - **Boundary Conditions**: {boundaries_12}
+    - **Implementation Example**: `{implementation_12}`
+
+    ---
+
+    ### Q7: "What is relative?" (Comparison Features)
+
+    **Concept**: {relative_feature_1_name}
+    - **Sample Fields Used**: fields_used_13
+    - **Definition**: {definition_13}
+    - **Why This Feature**: {why_13}
+    - **Logical Meaning**: {logical_meaning_13}
+    - **Directionality**: {directionality_13}
+    - **Boundary Conditions**: {boundaries_13}
+    - **Implementation Example**: `{implementation_13}`
+
+    **Concept**: {relative_feature_2_name}
+    - **Sample Fields Used**: fields_used_14
+    - **Definition**: {definition_14}
+    - **Why This Feature**: {why_14}
+    - **Logical Meaning**: {logical_meaning_14}
+    - **Directionality**: {directionality_14}
+    - **Boundary Conditions**: {boundaries_14}
+    - **Implementation Example**: `{implementation_14}`
+
+    ---
+
+    ### Q8: "What is essential?" (Essence Features)
+
+    **Concept**: {essence_feature_1_name}
+    - **Sample Fields Used**: fields_used_15
+    - **Definition**: {definition_15}
+    - **Why This Feature**: {why_15}
+    - **Logical Meaning**: {logical_meaning_15}
+    - **Directionality**: {directionality_15}
+    - **Boundary Conditions**: {boundaries_15}
+    - **Implementation Example**: `{implementation_15}`
+
+    **Concept**: {essence_feature_2_name}
+    - **Sample Fields Used**: fields_used_16
+    - **Definition**: {definition_16}
+    - **Why This Feature**: {why_16}
+    - **Logical Meaning**: {logical_meaning_16}
+    - **Directionality**: {directionality_16}
+    - **Boundary Conditions**: {boundaries_16}
+    - **Implementation Example**: `{implementation_16}`
+
+    ---
+
+    ## Implementation Considerations
+
+    ### Data Quality Notes
+    - **Coverage**: {coverage_note}
+    - **Timeliness**: {timeliness_note}
+    - **Accuracy**: {accuracy_note}
+    - **Potential Biases**: {bias_note}
+
+    ### Computational Complexity
+    - **Lightweight features**: {simple_features}
+    - **Medium complexity**: {medium_features}
+    - **Heavy computation**: {complex_features}
+
+    ### Recommended Prioritization
+
+    **Tier 1 (Immediate Implementation)**:
+    1. {priority_1_feature} - {priority_1_reason}
+    2. {priority_2_feature} - {priority_2_reason}
+    3. {priority_3_feature} - {priority_3_reason}
+
+    **Tier 2 (Secondary Priority)**:
+    1. {priority_4_feature} - {priority_4_reason}
+    2. {priority_5_feature} - {priority_5_reason}
+
+    **Tier 3 (Requires Further Validation)**:
+    1. {priority_6_feature} - {priority_6_reason}
+
+    ---
+
+    ## Critical Questions for Further Exploration
+
+    ### Unanswered Questions:
+    1. {unanswered_question_1}
+    2. {unanswered_question_2}
+    3. {unanswered_question_3}
+
+    ### Recommended Additional Data:
+    - {additional_data_1}
+    - {additional_data_2}
+    - {additional_data_3}
+
+    ### Assumptions to Challenge:
+    - {assumption_1}
+    - {assumption_2}
+    - {assumption_3}
+
+    ---
+
+    ## Methodology Notes
+
+    **Analysis Approach**: This report was generated by:
+    1. Deep field deconstruction to understand data essence
+    2. Question-driven feature generation (8 fundamental questions)
+    3. Logical validation of each feature concept
+    4. Transparent documentation of reasoning
+
+    **Design Principles**:
+    - Focus on logical meaning over conventional patterns
+    - Every feature must answer a specific question
+    - Clear documentation of "why" for each suggestion
+    - Emphasis on data understanding over prediction
+
+    ---
+
+    *Report generated: {generation_timestamp}*
+    *Analysis depth: Comprehensive field deconstruction + 8-question framework*
+    *Next steps: Implement Tier 1 features, validate assumptions, gather additional data as needed*
+
+
+
+## Core Analysis Principles
+
+1. **From Data Essence**: Start with what data truly means, not what it's traditionally used for
+2. **Autonomous Reasoning**: Skill performs all thinking, no user input required
+3. **Question-Driven**: Internal question bank guides feature generation
+4. **Meaning Over Patterns**: Prioritize logical meaning over conventional combinations
+5. **Transparency**: Show reasoning process in output
+
+## Example Output Structure
+
+When analyzing dataset 'BEME' (Balance Sheet and Market Data), the output would include:
+
+### Dataset Understanding
+**Fields Analyzed**: book_value, market_cap, book_to_market, etc.
+**Key Observations**: Dataset compares accounting values with market valuations
+
+### Field Deconstruction
+- **book_value**: Accountant's calculation of net asset value (quarterly, audited, historical cost-based)
+- **market_cap**: Market participants' valuation (continuous, forward-looking, sentiment-influenced)
+- **book_to_market**: Ratio comparing these two valuation perspectives
+
+### Feature Concepts Generated
+
+**From "What is stable?"**
+- "Market reevaluation stability": Rolling coefficient of variation of book_to_market
+- **Logic**: Measures whether market opinion is stable or volatile
+- **Meaning**: Stable values suggest consensus, volatile values suggest disagreement/uncertainty
+
+**From "What is changing?"**
+- "Value creation vs. market reevaluation decomposition": Separate book_value growth from market_cap growth
+- **Logic**: Distinguish fundamental value creation from market sentiment changes
+- **Meaning**: Which component drives changes in book_to_market?
+
+**From "What is combined?"**
+- "Intangible value proportion": (market_cap - book_value) / enterprise_value
+- **Logic**: Quantify proportion of value from intangibles (brand, growth, etc.)
+- **Meaning**: What percentage of valuation isn't captured on the balance sheet?
+
+**(Additional question-based features would follow...)**
+
+## Implementation Notes
+
+### The skill should:
+1. **Analyze first, then generate**: Fully understand dataset before proposing features
+2. **Show reasoning**: Explain why each feature concept makes sense
+3. **Be specific**: Reference actual field names and their characteristics
+4. **Be critical**: Question assumptions and identify limitations
+5. **Be creative**: Look beyond traditional financial metrics
+
+### The skill should NOT:
+1. **Ask users to think**: All thinking is internal to the skill
+2. **Provide generic templates**: Each analysis should be specific to the dataset
+3. **Rely on conventional wisdom**: Challenge traditional approaches
+4. **Output patterns without meaning**: Every suggestion must have clear logic
+
+## Quality Assurance
+
+**Self-Check Process:**
+- [ ] All fields analyzed, not just skimmed
+- [ ] Field meanings understood beyond descriptions
+- [ ] Multiple question types explored
+- [ ] Each feature has clear logical meaning
+- [ ] Reasoning is explicit, not implicit
+- [ ] Limitations are acknowledged
+- [ ] Output is dataset-specific, not generic
+
+**Validation Questions:**
+- Would this analysis help someone truly understand the data?
+- Are feature concepts novel yet meaningful?
+- Is the reasoning process transparent?
+- Does it avoid conventional thinking traps?
+
+---
+
+*This skill performs deep analysis of BRAIN datasets, generating meaningful feature engineering concepts based on data essence and logical reasoning.*