fableforge-agent-profiler 0.1.0__tar.gz

fableforge_agent_profiler-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 FableForge Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

fableforge_agent_profiler-0.1.0/PKG-INFO

@@ -0,0 +1,123 @@
+Metadata-Version: 2.4
+Name: fableforge-agent-profiler
+Version: 0.1.0
+Summary: Profile and classify agent behavior patterns from traces
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.0
+Requires-Dist: numpy>=1.24
+Requires-Dist: scikit-learn>=1.3
+Requires-Dist: rich>=13.0
+Requires-Dist: click>=8.0
+Requires-Dist: matplotlib>=3.7
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Dynamic: license-file
+
+# Agent Profiler
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/) [![Tests](https://img.shields.io/badge/tests-0-yellow.svg)](tests/)
+
+
+Profile and classify agent behavior patterns from traces using transition matrices and tool distributions.
+
+## Installation
+
+```bash
+pip install agent-profiler
+```
+
+## Quick Start
+
+### Profile a Session
+
+```bash
+# Profile and classify behavior
+aprof profile trace.jsonl
+
+# Save results to JSON
+aprof profile trace.jsonl -o results.json
+```
+
+### Classify Behavior
+
+```bash
+aprof classify trace.jsonl
+```
+
+### Generate Visualizations
+
+```bash
+aprof visualize trace.jsonl --output profile_chart.png
+```
+
+## Programming API
+
+```python
+from agent_profiler import AgentProfiler, BehaviorClassifier, ProfileVisualizer
+
+# Profile a session
+profiler = AgentProfiler()
+result = profiler.profile("trace.jsonl")
+print(f"Category: {result.category} (confidence: {result.confidence:.1%})")
+print(f"Tool distribution: {result.tool_distribution.tool_counts}")
+
+# Classify directly
+classifier = BehaviorClassifier()
+category, confidence, scores = classifier.classify(
+    edit_ratio=0.25, read_ratio=0.15, bash_ratio=0.30,
+    error_rate=0.4, error_recovery_rate=0.35
+)
+
+# Generate visualizations
+visualizer = ProfileVisualizer()
+visualizer.generate_profile_chart(result, output="profile.png")
+visualizer.generate_transition_heatmap("trace.jsonl", output="heatmap.png")
+visualizer.generate_tool_distribution_pie("trace.jsonl", output="tools.png")
+```
+
+## Behavior Categories
+
+| Category | Description | Key Indicators |
+|----------|-------------|----------------|
+| Debugging | Active debugging sessions | High Edit+Bash, many errors, recoveries |
+| Building | Feature development | High Write+Bash, low Read |
+| Exploring | Code investigation | High Read+Grep, low Edit |
+| Lost | Confused/circular behavior | Circular transitions, high Read |
+| Verifying | Change verification | Read after Edit, test runs |
+
+## License
+
+MIT
+
+## Ecosystem
+
+Part of the [FableForge](../) ecosystem — 21 open-source projects built from 210K real agent traces:
+
+| Project | Description |
+| --- | --- |
+| **[Anvil](../anvil)** | Self-verified coding agent |
+| **[VerifyLoop](../verifyloop)** | Plan→Execute→Verify→Recover framework |
+| **[ErrorRecovery](../error-recovery)** | Self-healing middleware (3,725 error patterns) |
+| **[FableForge-14B](../fableforge-14b)** | The fine-tuned 14B model (4-stage training) |
+| **[ShellWhisperer](../shell-whisperer)** | 1.5B edge agent (phone/RPi, 50ms) |
+| **[ReasonCritic](../reason-critic)** | Verification model (130 benchmark tasks) |
+| **[TraceCompiler](../trace-compiler)** | Compile traces → LoRA skills |
+| **[AgentRuntime](../agent-runtime)** | Persistent agent daemon (systemd for AI) |
+| **[AgentSwarm](../agent-swarm)** | Multi-agent from real trace transitions |
+| **[AgentTelemetry](../agent-telemetry)** | Datadog for agents (token tracking, costs) |
+| **[BenchAgent](../bench-agent)** | HumanEval for tool-use (107 tasks) |
+| **[AgentDev](../agent-dev)** | VSCode extension with verification |
+| **[TraceViz](../trace-viz)** | Trace replay visualizer (Next.js) |
+| **[AgentSkills](../agent-skills)** | npm for agent behaviors |
+| **[AgentCurriculum](../agent-curriculum)** | 5-stage progressive training |
+| **[AgentFuzzer](../agent-fuzzer)** | Adversarial testing for agents |
+| **[AgentConstitution](../agent-constitution)** | Safety guardrails from traces |
+| **[CostOptimizer](../cost-optimizer)** | Token cost reduction (50-80%) |
+| **[AgentProfiler](../agent-profiler)** | Behavioral fingerprinting |
+| **[TrajectoryDistiller](../trajectory-distiller)** | Trace→training data pipeline |
+| **[Fable5-Dataset](../fable5-dataset)** | HuggingFace dataset release |

fableforge_agent_profiler-0.1.0/README.md

@@ -0,0 +1,103 @@
+# Agent Profiler
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/) [![Tests](https://img.shields.io/badge/tests-0-yellow.svg)](tests/)
+
+
+Profile and classify agent behavior patterns from traces using transition matrices and tool distributions.
+
+## Installation
+
+```bash
+pip install agent-profiler
+```
+
+## Quick Start
+
+### Profile a Session
+
+```bash
+# Profile and classify behavior
+aprof profile trace.jsonl
+
+# Save results to JSON
+aprof profile trace.jsonl -o results.json
+```
+
+### Classify Behavior
+
+```bash
+aprof classify trace.jsonl
+```
+
+### Generate Visualizations
+
+```bash
+aprof visualize trace.jsonl --output profile_chart.png
+```
+
+## Programming API
+
+```python
+from agent_profiler import AgentProfiler, BehaviorClassifier, ProfileVisualizer
+
+# Profile a session
+profiler = AgentProfiler()
+result = profiler.profile("trace.jsonl")
+print(f"Category: {result.category} (confidence: {result.confidence:.1%})")
+print(f"Tool distribution: {result.tool_distribution.tool_counts}")
+
+# Classify directly
+classifier = BehaviorClassifier()
+category, confidence, scores = classifier.classify(
+    edit_ratio=0.25, read_ratio=0.15, bash_ratio=0.30,
+    error_rate=0.4, error_recovery_rate=0.35
+)
+
+# Generate visualizations
+visualizer = ProfileVisualizer()
+visualizer.generate_profile_chart(result, output="profile.png")
+visualizer.generate_transition_heatmap("trace.jsonl", output="heatmap.png")
+visualizer.generate_tool_distribution_pie("trace.jsonl", output="tools.png")
+```
+
+## Behavior Categories
+
+| Category | Description | Key Indicators |
+|----------|-------------|----------------|
+| Debugging | Active debugging sessions | High Edit+Bash, many errors, recoveries |
+| Building | Feature development | High Write+Bash, low Read |
+| Exploring | Code investigation | High Read+Grep, low Edit |
+| Lost | Confused/circular behavior | Circular transitions, high Read |
+| Verifying | Change verification | Read after Edit, test runs |
+
+## License
+
+MIT
+
+## Ecosystem
+
+Part of the [FableForge](../) ecosystem — 21 open-source projects built from 210K real agent traces:
+
+| Project | Description |
+| --- | --- |
+| **[Anvil](../anvil)** | Self-verified coding agent |
+| **[VerifyLoop](../verifyloop)** | Plan→Execute→Verify→Recover framework |
+| **[ErrorRecovery](../error-recovery)** | Self-healing middleware (3,725 error patterns) |
+| **[FableForge-14B](../fableforge-14b)** | The fine-tuned 14B model (4-stage training) |
+| **[ShellWhisperer](../shell-whisperer)** | 1.5B edge agent (phone/RPi, 50ms) |
+| **[ReasonCritic](../reason-critic)** | Verification model (130 benchmark tasks) |
+| **[TraceCompiler](../trace-compiler)** | Compile traces → LoRA skills |
+| **[AgentRuntime](../agent-runtime)** | Persistent agent daemon (systemd for AI) |
+| **[AgentSwarm](../agent-swarm)** | Multi-agent from real trace transitions |
+| **[AgentTelemetry](../agent-telemetry)** | Datadog for agents (token tracking, costs) |
+| **[BenchAgent](../bench-agent)** | HumanEval for tool-use (107 tasks) |
+| **[AgentDev](../agent-dev)** | VSCode extension with verification |
+| **[TraceViz](../trace-viz)** | Trace replay visualizer (Next.js) |
+| **[AgentSkills](../agent-skills)** | npm for agent behaviors |
+| **[AgentCurriculum](../agent-curriculum)** | 5-stage progressive training |
+| **[AgentFuzzer](../agent-fuzzer)** | Adversarial testing for agents |
+| **[AgentConstitution](../agent-constitution)** | Safety guardrails from traces |
+| **[CostOptimizer](../cost-optimizer)** | Token cost reduction (50-80%) |
+| **[AgentProfiler](../agent-profiler)** | Behavioral fingerprinting |
+| **[TrajectoryDistiller](../trajectory-distiller)** | Trace→training data pipeline |
+| **[Fable5-Dataset](../fable5-dataset)** | HuggingFace dataset release |

fableforge_agent_profiler-0.1.0/pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "fableforge-agent-profiler"
+version = "0.1.0"
+description = "Profile and classify agent behavior patterns from traces"
+readme = "README.md"
+requires-python = ">=3.10"
+license = {text = "MIT"}
+dependencies = [
+    "pydantic>=2.0",
+    "numpy>=1.24",
+    "scikit-learn>=1.3",
+    "rich>=13.0",
+    "click>=8.0",
+    "matplotlib>=3.7",
+]
+
+[project.scripts]
+aprof = "agent_profiler.cli:cli"
+
+[project.optional-dependencies]
+dev = ["pytest>=7.0", "pytest-cov", "ruff"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

fableforge_agent_profiler-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

fableforge_agent_profiler-0.1.0/src/agent_profiler/__init__.py

@@ -0,0 +1,13 @@
+"""Agent Profiler - Profile and classify agent behavior patterns."""
+
+from agent_profiler.profiler import AgentProfiler, ProfileResult
+from agent_profiler.classifier import BehaviorClassifier
+from agent_profiler.visualizer import ProfileVisualizer
+
+__all__ = [
+    "AgentProfiler",
+    "ProfileResult",
+    "BehaviorClassifier",
+    "ProfileVisualizer",
+]
+__version__ = "0.1.0"

fableforge_agent_profiler-0.1.0/src/agent_profiler/classifier.py

@@ -0,0 +1,266 @@
+"""Behavior classification using pretrained profiles and transition matrices."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+import numpy as np
+
+
+@dataclass
+class BehaviorProfile:
+    """A predefined behavior profile template."""
+
+    name: str
+    description: str
+    edit_weight: float = 0.0
+    read_weight: float = 0.0
+    grep_weight: float = 0.0
+    bash_weight: float = 0.0
+    write_weight: float = 0.0
+    error_rate_weight: float = 0.0
+    error_recovery_weight: float = 0.0
+    circular_weight: float = 0.0
+    entropy_weight: float = 0.0
+    min_turns: int = 0
+
+
+class DebuggingProfile(BehaviorProfile):
+    """Profile for debugging sessions: high edit+bash, many errors, recoveries."""
+
+    def __init__(self) -> None:
+        super().__init__(
+            name="debugging",
+            description="Active debugging with edits and error recovery loops",
+            edit_weight=0.25,
+            read_weight=0.15,
+            grep_weight=0.10,
+            bash_weight=0.30,
+            write_weight=0.05,
+            error_rate_weight=0.40,
+            error_recovery_weight=0.35,
+            circular_weight=0.20,
+            entropy_weight=0.15,
+            min_turns=5,
+        )
+
+
+class BuildingProfile(BehaviorProfile):
+    """Profile for productive building: high write+bash, low read."""
+
+    def __init__(self) -> None:
+        super().__init__(
+            name="building",
+            description="Active feature development with writes and executions",
+            edit_weight=0.15,
+            read_weight=0.10,
+            grep_weight=0.05,
+            bash_weight=0.25,
+            write_weight=0.30,
+            error_rate_weight=-0.10,
+            error_recovery_weight=0.10,
+            circular_weight=0.05,
+            entropy_weight=0.20,
+            min_turns=3,
+        )
+
+
+class ExploringProfile(BehaviorProfile):
+    """Profile for exploration: high read+grep, low edit."""
+
+    def __init__(self) -> None:
+        super().__init__(
+            name="exploring",
+            description="Code exploration with reads and searches, minimal edits",
+            edit_weight=0.05,
+            read_weight=0.35,
+            grep_weight=0.30,
+            bash_weight=0.10,
+            write_weight=0.02,
+            error_rate_weight=-0.20,
+            error_recovery_weight=0.05,
+            circular_weight=0.10,
+            entropy_weight=0.25,
+            min_turns=3,
+        )
+
+
+class LostProfile(BehaviorProfile):
+    """Profile for lost/confused sessions: circular transitions, high read."""
+
+    def __init__(self) -> None:
+        super().__init__(
+            name="lost",
+            description="Confused or circular behavior, reading without progress",
+            edit_weight=0.05,
+            read_weight=0.35,
+            grep_weight=0.15,
+            bash_weight=0.05,
+            write_weight=0.02,
+            error_rate_weight=0.10,
+            error_recovery_weight=-0.10,
+            circular_weight=0.40,
+            entropy_weight=0.10,
+            min_turns=5,
+        )
+
+
+class VerifyingProfile(BehaviorProfile):
+    """Profile for verification: read after edit, test runs."""
+
+    def __init__(self) -> None:
+        super().__init__(
+            name="verifying",
+            description="Verifying changes with reads after edits and test execution",
+            edit_weight=0.20,
+            read_weight=0.30,
+            grep_weight=0.10,
+            bash_weight=0.25,
+            write_weight=0.05,
+            error_rate_weight=-0.05,
+            error_recovery_weight=0.25,
+            circular_weight=0.15,
+            entropy_weight=0.15,
+            min_turns=4,
+        )
+
+
+class BehaviorClassifier:
+    """Classify agent behavior using pretrained profiles and scoring."""
+
+    def __init__(self) -> None:
+        self.profiles: dict[str, BehaviorProfile] = {
+            "debugging": DebuggingProfile(),
+            "building": BuildingProfile(),
+            "exploring": ExploringProfile(),
+            "lost": LostProfile(),
+            "verifying": VerifyingProfile(),
+        }
+
+    def compute_scores(
+        self,
+        edit_ratio: float = 0.0,
+        read_ratio: float = 0.0,
+        grep_ratio: float = 0.0,
+        bash_ratio: float = 0.0,
+        write_ratio: float = 0.0,
+        error_rate: float = 0.0,
+        error_recovery_rate: float = 0.0,
+        circular_ratio: float = 0.0,
+        entropy: float = 0.0,
+        num_turns: int = 0,
+    ) -> dict[str, float]:
+        """Compute profile scores for each behavior category.
+
+        Args:
+            edit_ratio: Ratio of edit tool calls.
+            read_ratio: Ratio of read tool calls.
+            grep_ratio: Ratio of grep/search tool calls.
+            bash_ratio: Ratio of bash/shell tool calls.
+            write_ratio: Ratio of write tool calls.
+            error_rate: Rate of errors in the session.
+            error_recovery_rate: Rate of error recovery.
+            circular_ratio: Ratio of circular tool transitions.
+            entropy: Entropy of tool distribution.
+            num_turns: Number of turns in the session.
+
+        Returns:
+            Dict mapping category name to confidence score (0-1).
+        """
+        scores: dict[str, float] = {}
+
+        features = {
+            "edit": edit_ratio,
+            "read": read_ratio,
+            "grep": grep_ratio,
+            "bash": bash_ratio,
+            "write": write_ratio,
+            "error_rate": error_rate,
+            "error_recovery": error_recovery_rate,
+            "circular": circular_ratio,
+            "entropy": entropy,
+        }
+
+        for name, profile in self.profiles.items():
+            profile_features = {
+                "edit": profile.edit_weight,
+                "read": profile.read_weight,
+                "grep": profile.grep_weight,
+                "bash": profile.bash_weight,
+                "write": profile.write_weight,
+                "error_rate": profile.error_rate_weight,
+                "error_recovery": profile.error_recovery_weight,
+                "circular": profile.circular_weight,
+                "entropy": profile.entropy_weight,
+            }
+
+            dot_product = sum(
+                features.get(k, 0.0) * profile_features.get(k, 0.0)
+                for k in set(list(features.keys()) + list(profile_features.keys()))
+            )
+
+            obs_vec = np.array([features.get(k, 0.0) for k in sorted(features.keys())])
+            prof_vec = np.array([profile_features.get(k, 0.0) for k in sorted(features.keys())])
+
+            obs_norm = np.linalg.norm(obs_vec)
+            prof_norm = np.linalg.norm(prof_vec)
+
+            if obs_norm > 0 and prof_norm > 0:
+                cosine_sim = float(np.dot(obs_vec, prof_vec) / (obs_norm * prof_norm))
+            else:
+                cosine_sim = 0.0
+
+            score = (dot_product + (cosine_sim + 1) / 2) / 2.0
+
+            if num_turns < profile.min_turns:
+                score *= 0.5
+
+            scores[name] = max(0.0, min(1.0, score))
+
+        total = sum(scores.values())
+        if total > 0:
+            scores = {k: v / total for k, v in scores.items()}
+
+        return scores
+
+    def classify(
+        self,
+        edit_ratio: float = 0.0,
+        read_ratio: float = 0.0,
+        grep_ratio: float = 0.0,
+        bash_ratio: float = 0.0,
+        write_ratio: float = 0.0,
+        error_rate: float = 0.0,
+        error_recovery_rate: float = 0.0,
+        circular_ratio: float = 0.0,
+        entropy: float = 0.0,
+        num_turns: int = 0,
+    ) -> tuple[str, float, dict[str, float]]:
+        """Classify behavior into a category.
+
+        Returns:
+            Tuple of (category, confidence, all_scores).
+        """
+        scores = self.compute_scores(
+            edit_ratio=edit_ratio,
+            read_ratio=read_ratio,
+            grep_ratio=grep_ratio,
+            bash_ratio=bash_ratio,
+            write_ratio=write_ratio,
+            error_rate=error_rate,
+            error_recovery_rate=error_recovery_rate,
+            circular_ratio=circular_ratio,
+            entropy=entropy,
+            num_turns=num_turns,
+        )
+        best = max(scores, key=scores.get)  # type: ignore[arg-type]
+        return best, scores[best], scores
+
+    def get_profile(self, name: str) -> BehaviorProfile | None:
+        """Get a profile by name."""
+        return self.profiles.get(name)
+
+    def list_profiles(self) -> dict[str, str]:
+        """List all available profiles with descriptions."""
+        return {name: p.description for name, p in self.profiles.items()}