skillinfer 0.1.0__tar.gz

skillinfer-0.1.0/.github/workflows/docs.yml

@@ -0,0 +1,38 @@
+name: Deploy docs
+
+on:
+  push:
+    branches: [main]
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: pages
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install mkdocs-material pymdown-extensions
+      - run: mkdocs build --strict --site-dir _site
+      - uses: actions/upload-pages-artifact@v3
+        with:
+          path: _site
+
+  deploy:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - id: deployment
+        uses: actions/deploy-pages@v4

skillinfer-0.1.0/.gitignore

@@ -0,0 +1,14 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+*.egg
+.venv/
+examples/.data/
+site/
+paper/
+data/
+results/
+.DS_Store

skillinfer-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kostadin Devedzhiev
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

skillinfer-0.1.0/PKG-INFO

@@ -0,0 +1,236 @@
+Metadata-Version: 2.4
+Name: skillinfer
+Version: 0.1.0
+Summary: Infer a full skill profile from a few observations. Few-shot capability estimation for AI agents and humans.
+Project-URL: Homepage, https://github.com/kostadindev/skillinfer
+Project-URL: Documentation, https://kostadindev.github.io/skillinfer
+Project-URL: Repository, https://github.com/kostadindev/skillinfer
+Project-URL: Issues, https://github.com/kostadindev/skillinfer/issues
+Author-email: Kostadin Devedzhiev <kgd26@cam.ac.uk>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: ai-agents,bayesian-inference,capability-estimation,human-ai-orchestration,kalman-filter,skill-inference
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=2.0
+Requires-Dist: scikit-learn>=1.2
+Requires-Dist: scipy>=1.10
+Provides-Extra: dev
+Requires-Dist: matplotlib>=3.7; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
+Requires-Dist: pymdown-extensions>=10.0; extra == 'docs'
+Provides-Extra: viz
+Requires-Dist: matplotlib>=3.7; extra == 'viz'
+Description-Content-Type: text/markdown
+
+# skillinfer
+
+**Infer a full skill profile from a few observations.**
+
+Observe a few skills, predict the rest — with calibrated uncertainty. `skillinfer` learns how capabilities co-vary across a population and uses that structure to infer a full profile from partial observations.
+
+A closed-form Bayesian update — no training loop, no GPU. One matrix operation gives you the exact posterior. Under 1ms per update, scales to 1000+ skills.
+
+## Install
+
+```bash
+pip install skillinfer
+```
+
+## Quick start
+
+```python
+import skillinfer
+
+pop = skillinfer.datasets.onet()          # 894 occupations x 120 skills
+profile = pop.profile()                   # new entity, unknown
+profile.observe("Skill:Programming", 0.92)
+print(profile.predict())                  # predict all 120 skills
+```
+
+```
+                           feature   mean    std  ci_lower  ci_upper
+    Skill:Complex Problem Solving   0.81   0.17      0.47      1.00
+          Skill:Critical Thinking   0.73   0.15      0.43      1.00
+              Skill:Programming     0.92   0.01      0.90      0.93  ← observed
+                Skill:Mathematics   0.67   0.12      0.43      0.91
+         Ability:Static Strength   0.10   0.23      0.00      0.55  ← anti-correlated
+...
+[120 rows x 5 columns]
+```
+
+## How it works
+
+When you observe one skill, the Kalman update propagates to every other skill via the learned covariance:
+
+- Skills with **positive covariance** move in the same direction (observe high Programming → predict high Analytical Reasoning)
+- Skills with **negative covariance** move opposite (observe high Programming → predict low Static Strength)
+- **Independent skills** are unaffected
+
+The update is the standard closed-form Gaussian conditioning rule, and reported predictions are clipped to `[0, 1]` to match the population's natural scale.
+
+Each `observe()` call is O(K²) — one matrix-vector product. No iteration, no convergence.
+
+## Core API
+
+```python
+import skillinfer
+from skillinfer import Skill, Task
+
+# Build a population from any entity-feature matrix
+pop = skillinfer.Population.from_dataframe(df)
+
+# Create a profile and observe
+profile = pop.profile()
+profile.observe("math", 0.95)
+profile.observe_many({"code": 0.89, "writing": 0.70})
+
+# Predict with uncertainty
+profile.predict()                          # all skills, with CIs
+profile.predict("reasoning")               # single skill
+profile.most_uncertain(k=3)                # what to assess next
+
+# Match agents to tasks
+task = Task({"math": 1.0, "reasoning": 0.5})
+result = profile.match_score(task, threshold=0.8)
+
+# Rank a pool of agents
+ranking = skillinfer.rank_agents(task, profiles, threshold=0.8)
+
+# Summary statistics
+profile.summary(true_vector=ground_truth)  # MAE, RMSE, coverage, etc.
+pop.summary()                              # condition number, sparsity, etc.
+```
+
+## Built-in datasets
+
+Two preprocessed datasets ship with the package (~440 KB total):
+
+```python
+# O*NET 30.2 — U.S. Department of Labor
+# 894 occupations x 120 features (skills, knowledge, abilities)
+pop = skillinfer.datasets.onet()
+
+# ESCO v1.2.1 — European Commission
+# 2,999 occupations x 134 skill groups (binary)
+pop = skillinfer.datasets.esco()
+```
+
+| Dataset | Entities | Features | Scale | Source |
+|---------|----------|----------|-------|--------|
+| **O\*NET** | 894 occupations | 120 (35 skills, 33 knowledge, 52 abilities) | Continuous [0, 1] | [O\*NET 30.2](https://www.onetcenter.org/database.html), CC BY 4.0 |
+| **ESCO** | 2,999 occupations | 134 Level-2 skill groups | Binary {0, 1} | [ESCO v1.2.1](https://esco.ec.europa.eu/) |
+
+## Use cases
+
+| Domain | Observe | Predict |
+|--------|---------|---------|
+| **AI model selection** | 1-2 benchmark scores | All benchmarks + best model for a task |
+| **Human skill profiling** | A few task observations | Full occupational profile (120 skills) |
+| **Human-AI orchestration** | Partial evals for both | Who handles which subtask |
+| **Worker-task matching** | Known competencies | Fit for new roles and tasks |
+
+## LLM orchestration
+
+`skillinfer` profiles are structured context you feed to an LLM orchestrator alongside cost, latency, and business constraints. The LLM reasons about observed vs. inferred skills and applies natural language constraints that no scoring function could replicate:
+
+```python
+from openai import OpenAI
+
+# Build profiles from partial evaluations
+agents = {
+    "gpt-4o":     {"reasoning": 0.92, "code": 0.89},
+    "claude-3.5": {"reasoning": 0.90, "writing": 0.95},
+    "gemini-pro": {"math": 0.88, "code": 0.82},
+}
+profiles = {
+    name: pop.profile().observe_many(obs)
+    for name, obs in agents.items()
+}
+
+# Format as context for the orchestrator
+agent_context = ""
+for name, profile in profiles.items():
+    agent_context += f"\n{name}:\n"
+    for skill in ["math", "reasoning", "code"]:
+        pred = profile.predict(skill)
+        source = "observed" if pred["std"] < 0.01 else "inferred"
+        agent_context += f"  {skill}: {pred['mean']:.2f} ± {pred['std']:.2f} ({source})\n"
+
+# The LLM decides — not a scoring function
+client = OpenAI()
+response = client.chat.completions.create(
+    model="gpt-4o-mini",
+    messages=[{"role": "user", "content": f"Pick an agent for this math task.\n{agent_context}"}],
+)
+```
+
+## Export / import
+
+```python
+# Population
+pop.to_csv("population.csv")
+pop.to_parquet("population.parquet")
+pop = skillinfer.Population.from_csv("population.csv")
+pop = skillinfer.Population.from_parquet("population.parquet")
+
+# Profile
+profile.to_json("profile.json")
+restored = skillinfer.Profile.from_json("profile.json")
+
+d = profile.to_dict()   # plain dict, JSON-serialisable
+restored = skillinfer.Profile.from_dict(d)
+```
+
+## Visualization
+
+Requires `pip install skillinfer[viz]`.
+
+```python
+import skillinfer
+
+pop = skillinfer.datasets.onet()
+profile = pop.profile()
+profile.observe("Skill:Programming", 0.92)
+
+# Population charts
+skillinfer.visualization.correlation_heatmap(pop)     # clustered correlation matrix
+skillinfer.visualization.scree_plot(pop)               # PCA variance explained
+skillinfer.visualization.feature_distributions(pop)    # box plots by variance
+skillinfer.visualization.skill_embedding(pop)          # 2D PCA feature map
+skillinfer.visualization.convergence_curve(pop)        # MAE vs. observations
+
+# Profile charts
+skillinfer.visualization.posterior_profile(profile)    # predicted skills + uncertainty
+skillinfer.visualization.prediction_scatter(profile, true_vec)  # predicted vs. true
+skillinfer.visualization.uncertainty_waterfall(pop, observations)  # uncertainty per observation
+skillinfer.visualization.compare_profiles({"dev": dev, "nurse": nurse})  # side-by-side
+```
+
+## Documentation
+
+Full documentation at [kostadindev.github.io/skillinfer](https://kostadindev.github.io/skillinfer):
+
+- [Quickstart](https://kostadindev.github.io/skillinfer/getting-started/quickstart/)
+- [Tutorials](https://kostadindev.github.io/skillinfer/tutorials/llm-benchmarks/) — LLM benchmarks, human skills, ESCO, agent orchestration
+- [How It Works](https://kostadindev.github.io/skillinfer/how-it-works/kalman-update/) — Kalman update, covariance estimation, computational cost
+- [API Reference](https://kostadindev.github.io/skillinfer/api/population/) — Population, Profile, Datasets, Visualization
+
+## License
+
+MIT

skillinfer-0.1.0/README.md

@@ -0,0 +1,196 @@
+# skillinfer
+
+**Infer a full skill profile from a few observations.**
+
+Observe a few skills, predict the rest — with calibrated uncertainty. `skillinfer` learns how capabilities co-vary across a population and uses that structure to infer a full profile from partial observations.
+
+A closed-form Bayesian update — no training loop, no GPU. One matrix operation gives you the exact posterior. Under 1ms per update, scales to 1000+ skills.
+
+## Install
+
+```bash
+pip install skillinfer
+```
+
+## Quick start
+
+```python
+import skillinfer
+
+pop = skillinfer.datasets.onet()          # 894 occupations x 120 skills
+profile = pop.profile()                   # new entity, unknown
+profile.observe("Skill:Programming", 0.92)
+print(profile.predict())                  # predict all 120 skills
+```
+
+```
+                           feature   mean    std  ci_lower  ci_upper
+    Skill:Complex Problem Solving   0.81   0.17      0.47      1.00
+          Skill:Critical Thinking   0.73   0.15      0.43      1.00
+              Skill:Programming     0.92   0.01      0.90      0.93  ← observed
+                Skill:Mathematics   0.67   0.12      0.43      0.91
+         Ability:Static Strength   0.10   0.23      0.00      0.55  ← anti-correlated
+...
+[120 rows x 5 columns]
+```
+
+## How it works
+
+When you observe one skill, the Kalman update propagates to every other skill via the learned covariance:
+
+- Skills with **positive covariance** move in the same direction (observe high Programming → predict high Analytical Reasoning)
+- Skills with **negative covariance** move opposite (observe high Programming → predict low Static Strength)
+- **Independent skills** are unaffected
+
+The update is the standard closed-form Gaussian conditioning rule, and reported predictions are clipped to `[0, 1]` to match the population's natural scale.
+
+Each `observe()` call is O(K²) — one matrix-vector product. No iteration, no convergence.
+
+## Core API
+
+```python
+import skillinfer
+from skillinfer import Skill, Task
+
+# Build a population from any entity-feature matrix
+pop = skillinfer.Population.from_dataframe(df)
+
+# Create a profile and observe
+profile = pop.profile()
+profile.observe("math", 0.95)
+profile.observe_many({"code": 0.89, "writing": 0.70})
+
+# Predict with uncertainty
+profile.predict()                          # all skills, with CIs
+profile.predict("reasoning")               # single skill
+profile.most_uncertain(k=3)                # what to assess next
+
+# Match agents to tasks
+task = Task({"math": 1.0, "reasoning": 0.5})
+result = profile.match_score(task, threshold=0.8)
+
+# Rank a pool of agents
+ranking = skillinfer.rank_agents(task, profiles, threshold=0.8)
+
+# Summary statistics
+profile.summary(true_vector=ground_truth)  # MAE, RMSE, coverage, etc.
+pop.summary()                              # condition number, sparsity, etc.
+```
+
+## Built-in datasets
+
+Two preprocessed datasets ship with the package (~440 KB total):
+
+```python
+# O*NET 30.2 — U.S. Department of Labor
+# 894 occupations x 120 features (skills, knowledge, abilities)
+pop = skillinfer.datasets.onet()
+
+# ESCO v1.2.1 — European Commission
+# 2,999 occupations x 134 skill groups (binary)
+pop = skillinfer.datasets.esco()
+```
+
+| Dataset | Entities | Features | Scale | Source |
+|---------|----------|----------|-------|--------|
+| **O\*NET** | 894 occupations | 120 (35 skills, 33 knowledge, 52 abilities) | Continuous [0, 1] | [O\*NET 30.2](https://www.onetcenter.org/database.html), CC BY 4.0 |
+| **ESCO** | 2,999 occupations | 134 Level-2 skill groups | Binary {0, 1} | [ESCO v1.2.1](https://esco.ec.europa.eu/) |
+
+## Use cases
+
+| Domain | Observe | Predict |
+|--------|---------|---------|
+| **AI model selection** | 1-2 benchmark scores | All benchmarks + best model for a task |
+| **Human skill profiling** | A few task observations | Full occupational profile (120 skills) |
+| **Human-AI orchestration** | Partial evals for both | Who handles which subtask |
+| **Worker-task matching** | Known competencies | Fit for new roles and tasks |
+
+## LLM orchestration
+
+`skillinfer` profiles are structured context you feed to an LLM orchestrator alongside cost, latency, and business constraints. The LLM reasons about observed vs. inferred skills and applies natural language constraints that no scoring function could replicate:
+
+```python
+from openai import OpenAI
+
+# Build profiles from partial evaluations
+agents = {
+    "gpt-4o":     {"reasoning": 0.92, "code": 0.89},
+    "claude-3.5": {"reasoning": 0.90, "writing": 0.95},
+    "gemini-pro": {"math": 0.88, "code": 0.82},
+}
+profiles = {
+    name: pop.profile().observe_many(obs)
+    for name, obs in agents.items()
+}
+
+# Format as context for the orchestrator
+agent_context = ""
+for name, profile in profiles.items():
+    agent_context += f"\n{name}:\n"
+    for skill in ["math", "reasoning", "code"]:
+        pred = profile.predict(skill)
+        source = "observed" if pred["std"] < 0.01 else "inferred"
+        agent_context += f"  {skill}: {pred['mean']:.2f} ± {pred['std']:.2f} ({source})\n"
+
+# The LLM decides — not a scoring function
+client = OpenAI()
+response = client.chat.completions.create(
+    model="gpt-4o-mini",
+    messages=[{"role": "user", "content": f"Pick an agent for this math task.\n{agent_context}"}],
+)
+```
+
+## Export / import
+
+```python
+# Population
+pop.to_csv("population.csv")
+pop.to_parquet("population.parquet")
+pop = skillinfer.Population.from_csv("population.csv")
+pop = skillinfer.Population.from_parquet("population.parquet")
+
+# Profile
+profile.to_json("profile.json")
+restored = skillinfer.Profile.from_json("profile.json")
+
+d = profile.to_dict()   # plain dict, JSON-serialisable
+restored = skillinfer.Profile.from_dict(d)
+```
+
+## Visualization
+
+Requires `pip install skillinfer[viz]`.
+
+```python
+import skillinfer
+
+pop = skillinfer.datasets.onet()
+profile = pop.profile()
+profile.observe("Skill:Programming", 0.92)
+
+# Population charts
+skillinfer.visualization.correlation_heatmap(pop)     # clustered correlation matrix
+skillinfer.visualization.scree_plot(pop)               # PCA variance explained
+skillinfer.visualization.feature_distributions(pop)    # box plots by variance
+skillinfer.visualization.skill_embedding(pop)          # 2D PCA feature map
+skillinfer.visualization.convergence_curve(pop)        # MAE vs. observations
+
+# Profile charts
+skillinfer.visualization.posterior_profile(profile)    # predicted skills + uncertainty
+skillinfer.visualization.prediction_scatter(profile, true_vec)  # predicted vs. true
+skillinfer.visualization.uncertainty_waterfall(pop, observations)  # uncertainty per observation
+skillinfer.visualization.compare_profiles({"dev": dev, "nurse": nurse})  # side-by-side
+```
+
+## Documentation
+
+Full documentation at [kostadindev.github.io/skillinfer](https://kostadindev.github.io/skillinfer):
+
+- [Quickstart](https://kostadindev.github.io/skillinfer/getting-started/quickstart/)
+- [Tutorials](https://kostadindev.github.io/skillinfer/tutorials/llm-benchmarks/) — LLM benchmarks, human skills, ESCO, agent orchestration
+- [How It Works](https://kostadindev.github.io/skillinfer/how-it-works/kalman-update/) — Kalman update, covariance estimation, computational cost
+- [API Reference](https://kostadindev.github.io/skillinfer/api/population/) — Population, Profile, Datasets, Visualization
+
+## License
+
+MIT

skillinfer-0.1.0/TODOS.md

@@ -0,0 +1,15 @@
+# TODOS — skillinfer
+
+## v0.2.0 Roadmap
+
+- [ ] **Active Evaluation Scheduler** (`next_eval()`): Multi-entity, budget-constrained evaluation scheduling. Given N agents and a budget of M benchmark runs, decide which (agent, benchmark) cell to fill next, maximizing information gain via the Kalman posterior covariance.
+- [ ] **Thompson sampling** in `rank_agents()`: `metric="thompson"` draws from the posterior for exploration. Requires handling near-singular covariance via `scipy.stats.multivariate_normal(allow_singular=True)` or Cholesky with jitter.
+- [ ] **Process noise**: Time-varying capabilities for agents that improve/degrade over time. Add Q matrix to the Kalman update for non-stationary features.
+- [ ] **GP baseline comparison**: Benchmark against Gaussian Processes for K=6 to honestly evaluate the accuracy/speed tradeoff vs GPs.
+
+## Nice-to-haves
+
+- [ ] Cache PCA in `Taxonomy.__str__()` to avoid recomputing on every print
+- [ ] Fuzzy match suggestions in error messages ("did you mean 'MATH Lvl 5'?")
+- [ ] `Taxonomy.new_states(names)` helper for batch state creation
+- [ ] Probit/ordinal link for binary/ordinal data (O*NET, student assessments)

skillinfer-0.1.0/docs/api/datasets.md

@@ -0,0 +1,168 @@
+# Datasets
+
+Built-in datasets that return a `Population` ready for profiling. No downloads, no preprocessing — just import and go.
+
+```python
+import skillinfer
+
+pop = skillinfer.datasets.onet()      # or skillinfer.datasets.esco()
+profile = pop.profile()
+profile.observe("Skill:Programming", 0.92)
+print(profile.predict())
+```
+
+Both datasets are shipped as compressed Parquet files inside the package (~440 KB total).
+
+---
+
+## `onet()`
+
+**O\*NET 30.2** — the U.S. Department of Labor's occupational information network.
+
+O\*NET is the most comprehensive public database of occupational skill requirements. It describes what workers in each occupation need to know and be able to do, based on surveys of incumbent workers and occupational analysts.
+
+```python
+pop = skillinfer.datasets.onet()
+print(pop)
+# Population(894 entities x 120 skills, shrinkage=0.0054)
+```
+
+### What's in it
+
+| | |
+|---|---|
+| **Entities** | 894 occupations (e.g., "Software Developers", "Registered Nurses", "Chief Executives") |
+| **Features** | 120 total: 35 skills, 33 knowledge areas, 52 abilities |
+| **Scale** | Continuous importance ratings, normalised to [0, 1] |
+| **Source** | [O\*NET 30.2](https://www.onetcenter.org/database.html), U.S. Department of Labor / ETA |
+| **License** | CC BY 4.0 |
+
+### Feature categories
+
+Features are prefixed by category:
+
+- **`Skill:`** — learned capabilities (e.g., `Skill:Programming`, `Skill:Critical Thinking`, `Skill:Writing`)
+- **`Knowledge:`** — domain knowledge (e.g., `Knowledge:Mathematics`, `Knowledge:Computers and Electronics`)
+- **`Ability:`** — enduring attributes (e.g., `Ability:Deductive Reasoning`, `Ability:Static Strength`, `Ability:Manual Dexterity`)
+
+### How it was preprocessed
+
+The raw [O\*NET 30.2 database](https://www.onetcenter.org/dl_files/database/db_30_2_text.zip) contains multiple scales per feature (importance, level, relevance). We extract:
+
+1. **[Skills.txt](https://www.onetcenter.org/dictionary/30.2/text/Skills.html)**, **[Knowledge.txt](https://www.onetcenter.org/dictionary/30.2/text/Knowledge.html)**, **[Abilities.txt](https://www.onetcenter.org/dictionary/30.2/text/Abilities.html)** from the O\*NET 30.2 database
+2. Filter to the **Importance** scale (`Scale ID = "IM"`), which rates each feature on a 1–5 scale
+3. Drop rows marked as suppressed (`Recommend Suppress = "Y"`)
+4. Pivot to an **occupation × feature** matrix (894 × 120)
+5. Normalise each column to **[0, 1]** using min-max scaling
+6. Replace O\*NET SOC codes with human-readable occupation titles
+
+### Population statistics
+
+| Statistic | Value |
+|-----------|-------|
+| Mean feature value | 0.385 |
+| Feature std | 0.228 |
+| Density (non-zero entries) | 92.2% |
+| Ledoit-Wolf shrinkage | 0.0054 |
+| Condition number | 1,884 |
+| Effective dimensions (90% var) | ~15 |
+| Mean \|correlation\| | 0.336 |
+| Correlation sparsity (<0.1) | 14.8% |
+
+The high mean correlation (0.336) and low sparsity (14.8%) mean most features are correlated — the population has rich transfer structure. Observing a few skills tells you a lot about the rest.
+
+### Example
+
+```python
+pop = skillinfer.datasets.onet()
+
+profile = pop.profile()
+profile.observe("Skill:Programming", 0.92)
+profile.observe("Skill:Critical Thinking", 0.85)
+print(profile.predict())
+
+# Use a specific occupation as prior
+profile = pop.profile(prior_entity="Software Developers")
+```
+
+**Parameters**
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `normalize` | `bool` | `False` | Re-normalise columns to [0, 1]. Data is already normalised, so this is rarely needed. |
+
+---
+
+## `esco()`
+
+**ESCO v1.2.1** — the European Commission's taxonomy of Skills, Competences, Qualifications and Occupations.
+
+ESCO is an independently curated taxonomy maintained by EU expert panels. Where O\*NET uses continuous importance ratings from U.S. surveys, ESCO uses binary skill assignments curated by European domain experts. This makes it a strong cross-validation target — if `skillinfer` works on both, the method generalises beyond any single taxonomy.
+
+```python
+pop = skillinfer.datasets.esco()
+print(pop)
+# Population(2999 entities x 134 skills, shrinkage=0.0211)
+```
+
+### What's in it
+
+| | |
+|---|---|
+| **Entities** | 2,999 occupations (e.g., "technical director", "registered nurse", "software developer") |
+| **Features** | 134 Level-2 skill groups |
+| **Scale** | Binary (1 = occupation requires at least one essential skill in that group, 0 = does not) |
+| **Source** | [ESCO v1.2.1](https://esco.ec.europa.eu/), European Commission |
+
+### How it was preprocessed
+
+The [ESCO classification](https://esco.ec.europa.eu/en/use-esco/download) assigns ~13,000 individual skills to occupations. We aggregate to a manageable matrix:
+
+1. Load **[occupationSkillRelations_en.csv](https://esco.ec.europa.eu/en/use-esco/download)** — maps occupations to individual skills with relation types (essential/optional)
+2. Filter to **essential** relations only
+3. Walk each skill up the **[ESCO skill hierarchy](https://esco.ec.europa.eu/en/about-esco/what-does-esco-cover/skills-and-competences)** (via `broaderRelationsSkillPillar_en.csv` and `skillsHierarchy_en.csv`) to its **Level-2 skill group** ancestor
+4. Build a binary **occupation × skill-group** matrix: 1 if the occupation has at least one essential skill in that group, 0 otherwise
+5. Replace skill group URIs with human-readable **preferred labels** from the hierarchy
+6. Drop occupations with fewer than 5 skill groups (too sparse to be informative)
+7. Replace occupation URIs with human-readable titles from **occupations_en.csv**
+
+### Population statistics
+
+| Statistic | Value |
+|-----------|-------|
+| Density (fraction of 1s) | 10.1% |
+| Ledoit-Wolf shrinkage | 0.0211 |
+| Condition number | 468 |
+| Effective dimensions (90% var) | ~15 |
+| Mean \|correlation\| | 0.055 |
+| Correlation sparsity (<0.1) | 84.6% |
+
+ESCO is much sparser than O\*NET — only 10% of entries are 1, and 84.6% of feature correlations are near zero. The covariance structure is concentrated in a few meaningful clusters (e.g., healthcare skills co-occur, IT skills co-occur), with most skill groups being independent. Transfer is still valuable but more targeted.
+
+### Key differences from O\*NET
+
+| | O\*NET | ESCO |
+|---|--------|------|
+| **Source** | U.S. Department of Labor surveys | EU expert panel curation |
+| **Scale** | Continuous [0, 1] | Binary {0, 1} |
+| **Features** | 120 (skills + knowledge + abilities) | 134 (skill groups only) |
+| **Entities** | 894 occupations | 2,999 occupations |
+| **Correlation structure** | Dense (mean \|r\| = 0.34) | Sparse (mean \|r\| = 0.06) |
+| **Best for** | Rich skill profiling, continuous predictions | Cross-validation, binary classification tasks |
+
+### Example
+
+```python
+pop = skillinfer.datasets.esco()
+
+profile = pop.profile()
+profile.observe("education", 1.0)
+profile.observe("teaching and training", 1.0)
+print(profile.predict())
+```
+
+**Parameters**
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `normalize` | `bool` | `False` | Normalise columns to [0, 1]. Data is binary, so this is rarely needed. |