eduloggen 1.4.1__tar.gz

eduloggen-1.4.1/.gitignore

@@ -0,0 +1,227 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#   Usually these files are written by a python script from a template
+#   before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+# Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+# uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+# poetry.lock
+# poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+# pdm.lock
+# pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+# pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# Redis
+*.rdb
+*.aof
+*.pid
+
+# RabbitMQ
+mnesia/
+rabbitmq/
+rabbitmq-data/
+
+# ActiveMQ
+activemq-data/
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#   JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#   be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#   and can be added to the global gitignore or merged into this file.  For a more nuclear
+#   option (not recommended) you can uncomment the following to ignore the entire idea folder.
+# .idea/
+
+# Abstra
+#   Abstra is an AI-powered process automation framework.
+#   Ignore directories containing user credentials, local state, and settings.
+#   Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#   Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore
+#   that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#   and can be added to the global gitignore or merged into this file. However, if you prefer,
+#   you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+# Temporary file for partial code execution
+tempCodeRunnerFile.py
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# Streamlit
+.streamlit/secrets.toml
+
+# macOS Finder metadata
+.DS_Store
+
+# Real datasets (download separately; never commit)
+/OULAD/
+studies/**/data/
+studies/**/out/
+studies/**/run.log

eduloggen-1.4.1/CHANGELOG.md

@@ -0,0 +1,278 @@
+# Changelog
+
+All notable changes to EduLogGen are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [1.4.1] - 2026-10-09
+
+Ready for public release: PyPI publishing, citation metadata, an OULAD case
+study, and more robust automatic profiles on real data.
+
+### Added
+
+- Publishing: PyPI release workflow (trusted publishing, triggered by a
+  version tag, with a GitHub release from the changelog), documentation
+  deployment to GitHub Pages, `CITATION.cff`, `SECURITY.md`, and
+  `CODE_OF_CONDUCT.md`
+- Case study on the Open University Learning Analytics Dataset
+  (`studies/oulad/`, docs page "Case study: OULAD") and a SoftwareX
+  manuscript draft (`paper/softwarex/`)
+
+### Changed
+
+- README: `pip install eduloggen`, a citation section, and absolute links so
+  they work on PyPI
+
+### Fixed
+
+- Markov back-off now says when every session has a single event, instead
+  of "reducing Markov order from 1 to 1"
+- Automatic profiles on real data: event types below 1% of events are now
+  pooled into one `share:(other)` feature, and standardised features are
+  capped at ±3, so a handful of learners who used a rare activity no longer
+  form clusters of their own (found on OULAD, where k=4 produced clusters of
+  3 and 4 learners; now 42–80). `Standardizer` gains `clip` (default 3);
+  profile sets saved by 1.4.0 keep assigning without a cap.
+- `fit_profiles` no longer fails when a profile only has very short sessions
+  (e.g. learners active one day a week): per-profile models now lower the
+  Markov order instead (`on_insufficient_data: backoff`) unless you set
+  `on_insufficient_data` yourself.
+
+## [1.4.0] - 2026-10-09
+
+Level 2, milestone M3: behavioural profiles and profile mixtures, and
+generators that learn when sessions start.
+
+### Changed
+
+- Generators now learn **when** sessions start (generator version 1.1):
+  models record session starts per hour of day (UTC) and per day of the
+  training period, and generation draws starts from that learned calendar.
+  Synthetic sessions no longer start in hours or on days without real
+  activity (on the demo course, 45% of sessions previously started in hours
+  with no real sessions; now none do). Models fitted with earlier versions
+  lack these counts and generate exactly as before. An explicit `calendar`
+  still takes precedence; `start_time` moves the learned pattern and remains
+  the earliest possible start.
+
+### Added
+
+- Level 2, M3 — behavioural profiles (`eduloggen.scenarios`):
+  - `fit_profiles(mode="auto")`: seeded k-means++ (with restarts) on
+    standardised per-learner features (sessions, session length, duration,
+    success rate, event-type shares); profiles `profile_1..k` ordered by
+    size, with a table of distinguishing features; `ProfileSet.assign` maps
+    new (e.g. holdout) learners to the nearest training profile
+  - `fit_profiles(mode="provided")` with a `learner_id -> profile` mapping
+    (`load_profile_assignments` reads a CSV)
+  - `define_profiles` (manual): profiles defined by start and transition
+    probabilities, session lengths, sessions per learner, dwell times, and
+    activities, without real data; proportions alone are rejected
+  - one generator model per profile; profiles below `min_learners`
+    (default 5) are an error with guidance
+  - `generate_profiles`: mixture by share of synthetic learners, optional
+    per-profile controls and a shared calendar, ids remapped after mixing,
+    one learner-level `profile` annotation per synthetic learner
+  - `ProfileSet.save` / `load_profiles` with a content fingerprint
+- `evaluate_clustering` (`eduloggen.evaluation`): adjusted Rand index,
+  normalised mutual information, purity, and the contingency table
+- `SessionCalendar.daily`: optional per-day weights (used by learned
+  calendars; also available in calendar files)
+
+## [1.3.0] - 2026-10-09
+
+Fine-grained analysis: per activity, over time, per group, and a ranked view
+of where synthetic data differs from real data.
+
+### Added
+
+- Per-activity profiles (`activity_profiles`): share, session and learner
+  reach, start/end shares, repeat share, dwell-time distribution,
+  predecessors and successors, success rate, mean score, companions
+- Temporal analysis (`temporal_profile`, `deadline_effects`): timezone-aware
+  activity by hour and weekday, weekday x hour table, weekly trends from the
+  first session, daily counts, gaps between a learner's sessions, activity
+  ratio before deadlines
+- Stratified analysis (FR-A.7; `analyze_by`, `split_by`, `session_groups`):
+  by `course`, `week`, `weekday`, `hour`, `learner_group`, or
+  `metadata:<key>`, with each group's mix compared to the overall mix
+- Fine-grained validation (`compare_detailed`): ranked per-token,
+  per-transition (including invented and never-produced transitions),
+  per-length, per-hour, per-weekday, and per-group differences
+- CLI: `analyze --detail --by --groups --timezone --deadlines`,
+  `validate --detailed --by --groups --timezone`; plot `activity_heatmap`
+
+## [1.2.0] - 2026-10-09
+
+Level 2, milestone M2: run controlled experiments and prove each control
+took effect.
+
+### Added
+
+- Level 2, milestone M2: behavioural controls, course calendars, and
+  manipulation checks
+- `eduloggen.scenarios.Controls` / `apply_controls`: `event_weights`
+  (reweight transitions into a token; `0` removes it), `dwell_scale`,
+  `session_length` (scale or fixed), `sessions_per_learner` (mean or fixed);
+  controls produce a new model whose parameters record them
+- `eduloggen.generators.SessionCalendar`: period, timezone-aware hour and
+  weekday weights, deadline surges; `generate(..., calendar=...)` for every
+  generator; a learner's sessions never overlap and never start in
+  zero-weight hours
+- `manipulation_check`: compares a controlled sample with an uncontrolled
+  baseline (same model, seed, size) for every control, the calendar, and
+  anomalies; sample-size-aware tolerances for distributional checks
+- `ExperimentSettings` / `run_experiment` and CLI
+  `generate --experiment FILE` (controls, calendar, anomalies), writing
+  `manipulation_check.json` / `.md`
+- Generator capability tag `supports_event_weights`
+- Example `examples/configs/experiment.yaml`; experiment guide
+
+## [1.1.0] - 2026-10-09
+
+Level 2, milestone M1: benchmark anomaly detectors on synthetic data with
+labelled ground truth.
+
+### Added
+
+- Level 2, milestone M1 (`docs/04_EXPERIMENTAL_GENERATOR.md`): labelled
+  anomaly injection for benchmarking detectors
+- `eduloggen.scenarios`: five anomaly injectors — `event_frequency`,
+  `abnormal_timing`, `repetition`, `inactivity` (unusual but valid) and
+  `unexpected_transition` (invalid workflow); `inject_anomalies` with seeded,
+  disjoint session selection, one anomaly per session, no overlapping
+  sessions, and full id remapping so ids never reveal injected records;
+  `register_anomaly` for custom injectors
+- `Annotations` ground-truth table in `eduloggen.models`, stored as
+  `annotations.csv` in corpora with its own fingerprint (`read_annotations`)
+- `eduloggen.evaluation.evaluate_detection`: precision, recall, F1, false
+  positive rate, ROC-AUC, average precision, and recall per anomaly type and
+  per category
+- CLI: `generate --anomalies FILE` and `evaluate --corpus --predictions`
+- `privacy.remap_ids_with_mapping` returning the old-to-new id mapping
+- Example `examples/configs/anomalies.yaml`; detector-testing guide
+
+## [1.0.0] - 2026-10-09
+
+First stable release: the complete Level 1 log generator. Learn from real
+educational interaction logs, generate privacy-conscious synthetic sessions,
+validate them, compare generators, and plot the results, from Python or the
+`eduloggen` command line.
+
+### Added
+
+- `eduloggen.core` foundation package: exception hierarchy with machine-readable
+  codes, schema/artifact/report version constants, default `event_type`
+  vocabulary, immutable `RunContext`, and shared type aliases
+- `eduloggen.models` canonical domain model: immutable, validated `LogRecord`,
+  `Session`, and `Participant`; `Dataset` / `SyntheticDataset` snapshots with
+  cross-record checks, content fingerprints, and manifests; `EventVocabulary`
+  with preserve / map-to-other / reject policies; `Feature`
+- `eduloggen.io` ingestion and corpus I/O: streaming CSV/TSV, JSON Lines, and
+  Parquet readers and writers; `FieldMapping` with automatic casting,
+  timestamp formats and timezone policy, defaults, salted ID pseudonymization,
+  and metadata passthrough; `ingest()` with deduplication, sorting, strict
+  mode, and a value-free data quality report; atomic corpus directory
+  read/write with overwrite protection and fingerprint integrity checks
+- `parquet` optional extra (`pyarrow`)
+- `eduloggen.config`: strict, immutable `AppConfig` with sections from PRD §16.4;
+  YAML/TOML/JSON loading; precedence CLI > `EDULOGGEN_*` env allowlist > file >
+  defaults via `resolve_config`; config fingerprints; relative paths resolved
+  against the config file
+- `FieldMapping.from_file` for YAML/TOML/JSON mapping files
+- Example experiment config and field mapping under `examples/configs/`
+- `pyyaml` as the first runtime dependency
+- `eduloggen.utils`: versioned, order-independent seed derivation
+  (`derive_seed`, `make_rng`) and canonical JSON fingerprints
+- `eduloggen.privacy`: seeded remapping of learner/session/event ids
+  (default for generated data) and metadata key stripping
+- Architecture test enforcing the package dependency matrix (SAD §8.2)
+- `eduloggen.analysis`: `sessionize()` with explicit / idle-timeout /
+  composite strategies and configurable tokenization; order-k transition
+  counts with start padding and additive smoothing; n-grams and rare-n-gram
+  detection; inter-event, sojourn, and duration statistics; session feature
+  rows, learner profiles, corpus features, and a navigation graph;
+  `analyze()` returning an `AnalysisResult` with JSON and Markdown exports
+- `eduloggen.generators`: `BaseGenerator` contract (fit / generate / describe /
+  save / load) with shared population modelling (sessions per learner,
+  inter-session gaps, companion fields, courses) and seeded, id-remapped
+  output; `markov` (order-k with backoff and additive smoothing; empirical,
+  Poisson, or fixed lengths), `semi_markov` (per-token sojourn timing:
+  empirical, lognormal, gamma, exponential), and `independent` baseline;
+  generator registry; inspectable JSON model artifacts with integrity checks
+- `GeneratorModel` in `eduloggen.models`
+- `eduloggen.validation`: `validate()` comparing real and synthetic datasets
+  with 11 built-in metrics — `event_type_tvd`, `activity_jsd`,
+  `session_length_ks`, `session_duration_w1`, `interevent_time_ks`,
+  `bigram_tvd`, `transition_jsd`, `topn_path_overlap`, and privacy indicators
+  `exact_session_dup_rate`, `rare_ngram_replay_rate`, `nn_distance_p05`;
+  thresholds with pass / fail / info / skip statuses; `ValidationReport` with
+  JSON and Markdown output; metric registry for custom metrics
+- `eduloggen.datasets.demo_dataset`: seeded, fully synthetic demo course corpus
+- `eduloggen.benchmark`: protocol `session_fidelity_v1` with a learner-level
+  holdout split, multi-seed runs, mean/std aggregation, fit/generate timings,
+  a real-vs-real reference row, failure isolation per generator, and
+  `BenchmarkReport` JSON/Markdown output; `run_benchmark` in the API
+- CLI commands `benchmark` and `demo`
+- CI smoke job running the full pipeline and benchmark on the demo corpus
+- `eduloggen.visualization` (optional `viz` extra): event/activity frequency,
+  session length and duration, inter-event time (histogram and ECDF), and
+  transition heatmap comparisons; sampled session timelines; validation and
+  benchmark charts; colour-blind-safe palette; reproducible PNG/SVG/PDF export;
+  `plot_datasets` for the standard set; CLI `plot` command
+- `eduloggen.plugins`: entry-point discovery (`eduloggen.plugins.<kind>`) for
+  generators, metrics, readers, visualizers, and benchmark suites; contract
+  checks before registration; built-ins protected; broken plugins reported
+  without crashing (`strict=True` to fail); `register_plugin`,
+  `unregister_plugin`, `list_plugins`; CLI discovers at startup, `plugins`
+  shows sources, versions, and errors; run manifests record plugin versions
+- Registration hooks: `register_reader` (with file suffixes),
+  `register_plot`, `register_protocol`; `BUILTIN_*` name sets
+- Plugin author guide (`docs/plugins.md`)
+- Node-link transition graph (`plot_transition_graph`) and multi-step Sankey
+  pathway diagram (`plot_sankey`, `sankey_flows`), matplotlib only
+- `eduloggen.api` workflow façade (`load_config`, `ingest`, `load_dataset`,
+  `save_dataset`, `sessionize`, `analyze`, `fit_generator`, `generate`,
+  `validate`), re-exported from `import eduloggen`
+- CLI commands `ingest`, `sessionize`, `analyze`, `fit`, `generate`,
+  `validate`, `info`, `plugins` with global `--config`, `--seed`, `-v`,
+  `--quiet`, `--json-logs`, `--force`, `--run-id`; exit codes 0/1/2;
+  `run_manifest.json` in every output directory; `python -m eduloggen`
+- Documentation: privacy and responsible-use statement, end-to-end
+  getting-started guide, API reference pages for every package
+
+### Changed
+
+- Atomic output-directory writing moved to `eduloggen.utils.fs` and shared by
+  corpus and model artifacts
+- SAD §8.2: `validation` may depend on `analysis` (documented in the matrix)
+- Coverage gate raised from 80% to 95% (ADR-011)
+- `io.format` and `ingest(format=...)` accept plugin reader names
+- `eduloggen.cli` is now a package; the `eduloggen` console script entry
+  point is unchanged
+- `eduloggen.ingestion` now re-exports the ingest API from `eduloggen.io`
+
+## [0.1.0] - 2026-07-29
+
+### Added
+
+- Initial production-quality package skeleton (`src` layout)
+- Package metadata and build configuration via `pyproject.toml`
+- Tooling: Ruff, Black, MyPy, PyTest, pre-commit, MkDocs
+- GitHub Actions continuous integration workflow
+- Documented subpackage namespaces for ingestion, analysis, generators,
+  validation, visualization, and CLI
+- MIT license and Semantic Versioning policy
+
+[Unreleased]: https://github.com/anisbey1/EduLogGen/compare/v1.4.1...HEAD
+[1.4.1]: https://github.com/anisbey1/EduLogGen/compare/v1.4.0...v1.4.1
+[1.4.0]: https://github.com/anisbey1/EduLogGen/compare/v1.3.0...v1.4.0
+[1.3.0]: https://github.com/anisbey1/EduLogGen/compare/v1.2.0...v1.3.0
+[1.2.0]: https://github.com/anisbey1/EduLogGen/compare/v1.1.0...v1.2.0
+[1.1.0]: https://github.com/anisbey1/EduLogGen/compare/v1.0.0...v1.1.0
+[1.0.0]: https://github.com/anisbey1/EduLogGen/compare/v0.1.0...v1.0.0
+[0.1.0]: https://github.com/anisbey1/EduLogGen/releases/tag/v0.1.0

eduloggen-1.4.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 anisbey1
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.