ci-time-tracker 0.2.0__tar.gz

ci_time_tracker-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 ci-time-tracker contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ci_time_tracker-0.2.0/PKG-INFO

@@ -0,0 +1,561 @@
+Metadata-Version: 2.4
+Name: ci-time-tracker
+Version: 0.2.0
+Summary: Find where CI time and money go: cost, flaky steps and regressions from GitHub Actions, and a PR comment action
+Author: ci-time-tracker contributors
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/0xWhisp/ci-time-tracker
+Project-URL: Repository, https://github.com/0xWhisp/ci-time-tracker
+Keywords: ci,cd,github-actions,flaky-tests,ci-cost,devops
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Build Tools
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: PyYAML>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: hypothesis>=6.0; extra == "dev"
+Requires-Dist: ruff>=0.17; extra == "dev"
+Dynamic: license-file
+
+# ci-time-tracker
+
+[![CI](https://github.com/0xWhisp/ci-time-tracker/actions/workflows/ci.yml/badge.svg)](https://github.com/0xWhisp/ci-time-tracker/actions/workflows/ci.yml)
+
+Find where your CI time and money go. ci-time-tracker reads your GitHub Actions
+history and reports the steps that cost the most, the ones that are flaky, and
+the ones that got slower, with the commit where it started. Its GitHub Action
+comments on each pull request with how it changes CI time and cost.
+
+## Features
+
+- **Config Analysis**: Parse GitHub Actions, GitLab CI, and CircleCI configurations to extract pipeline structure
+- **Log Analysis**: Parse build logs to compute actual step durations and statistics
+- **Flaky Detection**: Flag steps that both failed and passed on the same commit,
+  including runs that only passed on a re-run
+- **Regression Detection**: Flag steps whose duration jumped and stayed up, and
+  name the commit where it started
+- **Cost and Time**: Rank steps by the time they consume, estimate cost per runner,
+  and separate queue time from execution time
+- **Pull Request Comments**: A GitHub Action that tells each pull request how it
+  changes CI time and cost, and whether its failures are in known flaky steps
+- **Multiple Output Formats**: Human-readable text or JSON for dashboard integration
+
+## Installation
+
+Requires Python 3.10+. Not yet published on PyPI; install from source:
+
+```bash
+git clone https://github.com/0xWhisp/ci-time-tracker.git
+cd ci-time-tracker
+pip install -e .
+```
+
+## Usage
+
+### Config Analysis Mode
+
+Analyze pipeline configuration files to understand structure and estimated runtimes:
+
+```bash
+# Analyze GitHub Actions workflow
+ci-time-tracker --config .github/workflows/ci.yml
+
+# Analyze with runtime estimates
+ci-time-tracker --config .gitlab-ci.yml --estimates estimates.json
+
+# Output as JSON
+ci-time-tracker --config .circleci/config.yml --format json
+
+# Save output to file
+ci-time-tracker --config .github/workflows/ci.yml --output report.txt
+```
+
+**Estimates File Format** (`estimates.json`):
+```json
+{
+  "Checkout": 5.0,
+  "Build": 120.0,
+  "Test": 300.0,
+  "Deploy": 60.0
+}
+```
+
+### GitHub Actions API Mode
+
+Analyze recent workflow runs straight from the GitHub Actions API, with no log
+files to collect. Timings come from the API, so nothing depends on parsing
+runner log text:
+
+```bash
+# Last 50 runs of every workflow in the repository
+ci-time-tracker --github acme/web
+
+# Last 200 runs of a single workflow on main
+ci-time-tracker --github acme/web --workflow ci.yml --branch main --last 200
+
+# JSON for a dashboard
+ci-time-tracker --github acme/web --format json
+```
+
+Set `GITHUB_TOKEN` (or `GH_TOKEN`) to reach private repositories and to raise the
+rate limit from 60 to 5000 requests per hour:
+
+```bash
+export GITHUB_TOKEN=$(gh auth token)
+```
+
+Each run costs one API request for its jobs, so completed runs are cached
+locally and later invocations only fetch what is new. The cache lives in the
+platform cache directory (override with `--cache-path` or the
+`CI_TIME_TRACKER_CACHE` environment variable), and `--no-cache` skips it.
+
+Steps are reported as `job / step`, the way the GitHub UI names them, so
+identically named steps in different jobs stay separate.
+
+### Log Analysis Mode
+
+Analyze build logs to compute actual durations and detect issues:
+
+```bash
+# Analyze single log file
+ci-time-tracker --logs build.log
+
+# Analyze directory of logs
+ci-time-tracker --logs logs/
+
+# Output to file
+ci-time-tracker --logs build.log --output report.json --format json
+```
+
+### Piping Input
+
+Both config and log modes support stdin input using `-`:
+
+```bash
+# Pipe config from stdin
+cat .github/workflows/ci.yml | ci-time-tracker --config -
+
+# Pipe log from stdin
+cat build.log | ci-time-tracker --logs -
+
+# Chain with other commands
+curl https://example.com/config.yml | ci-time-tracker --config - --format json
+```
+
+### Module Execution
+
+You can also run ci-time-tracker as a Python module:
+
+```bash
+python -m ci_time_tracker --config .github/workflows/ci.yml
+```
+
+## Output Example
+
+### Text Output (Log Mode)
+
+```
+==============================================================================================================
+CI Time Tracker Report (log mode)
+==============================================================================================================
+Generated: 2026-09-27 03:43:13
+Mode: log
+
+SUMMARY
+--------------------------------------------------------------------------------------------------------------
+  build_count: 10
+  step_count: 66
+  total_execution_seconds: 1h 05m
+  job_count: 35
+  total_queue_seconds: 2m 42s
+  queue_p50_seconds: 4.0s
+  queue_p90_seconds: 10.0s
+  estimated_cost_usd: $2.81
+  priced_jobs: 35
+  pricing_source: built-in defaults
+  flaky_detection: same-commit
+  regression_checked_steps: 12
+  regression_min_builds: 10
+  note: cost uses private-repository rates and GitHub's per-job rounding; public repos pay nothing for standard runners
+
+TOP TIME CONSUMERS
+--------------------------------------------------------------------------------------------------------------
+  1. build / Run tests                              45m 18s   69.2%    23 runs     $1.53
+  2. build / Install dependencies                    7m 08s   10.9%    23 runs     $0.30
+  3. build / Set up Python                           1m 43s    2.6%    23 runs     $0.08
+
+ISSUES
+--------------------------------------------------------------------------------------------------------------
+  [WARNING] REGRESSION: build / Install dependencies
+    Median 10.0s -> 28.0s (+180%, +18.0s)
+    Started at commit 3f9c2ab (build 9876543210, 2026-09-14)
+  [ERROR] FLAKY: integration / Run e2e tests
+    Failed and passed on the same commit in 2 of 17 commits
+    Passed on re-run after failing in 2 run(s)
+    e.g. 1a2b3c4, 9f8e7d6
+
+STEPS
+--------------------------------------------------------------------------------------------------------------
+Step Name                               Runs     Total      P50      P90      P95      P99     Cost Flags
+--------------------------------------------------------------------------------------------------------------
+build / Run tests                         23   45m 18s  112.00s  178.00s  178.00s  179.00s    $1.53
+build / Install dependencies              23    7m 08s   10.00s   28.00s   31.00s  115.00s    $0.30 REGR
+build / Set up Python                     23    1m 43s    2.00s    9.00s   44.00s   49.00s    $0.08
+... and 63 more steps (use --top 0 to show all)
+
+==============================================================================================================
+```
+
+Steps are ordered by the time they consume, because that is what a CI bill is
+made of. `--top N` limits the table (default 25, `0` shows everything). (The
+issues in this example are illustrative; the rest is real output for
+psf/requests.)
+
+### JSON Output
+
+```json
+{
+  "title": "CI Time Tracker Report (log mode)",
+  "mode": "log",
+  "generated_at": "2026-09-27T03:43:13",
+  "summary": {
+    "mode": "log",
+    "build_count": 10,
+    "step_count": 66,
+    "total_execution_seconds": 3918.0,
+    "job_count": 35,
+    "total_queue_seconds": 162.0,
+    "queue_p50_seconds": 4.0,
+    "queue_p90_seconds": 10.0,
+    "estimated_cost_usd": 2.808,
+    "priced_jobs": 35,
+    "pricing_source": "built-in defaults"
+  },
+  "steps": [
+    {
+      "name": "build / Run tests",
+      "execution_count": 23,
+      "success_count": 23,
+      "failure_count": 0,
+      "failure_rate": 0.0,
+      "total_duration": 2718.0,
+      "mean_duration": 118.17,
+      "p50": 112.0,
+      "p90": 178.0,
+      "p95": 178.0,
+      "p99": 179.0,
+      "is_regression": false,
+      "is_flaky": false,
+      "runner": "ubuntu-22.04",
+      "estimated_cost_usd": 1.5312,
+      "regression": null,
+      "flaky_evidence": null
+    }
+  ],
+  "issues": [
+    {
+      "type": "regression",
+      "step": "build / Install dependencies",
+      "severity": "warning",
+      "baseline_median": 10.0,
+      "current_median": 28.0,
+      "increase_seconds": 18.0,
+      "increase_pct": 1.8,
+      "baseline_builds": 14,
+      "current_builds": 9,
+      "started_at": {
+        "build_id": "9876543210",
+        "head_sha": "3f9c2ab",
+        "timestamp": "2026-09-14T10:02:11+00:00"
+      }
+    }
+  ]
+}
+```
+
+## Using It in CI
+
+### Run summary in Markdown
+
+`--format markdown` renders the report for a workflow run's summary page or a
+PR comment: headline numbers, the issues with their evidence, the top time
+consumers, and the full step table collapsed.
+
+```yaml
+- name: CI time report
+  run: |
+    pip install git+https://github.com/0xWhisp/ci-time-tracker.git
+    ci-time-tracker --github ${{ github.repository }} --workflow ci.yml       --last 100 --group-matrix --format markdown >> "$GITHUB_STEP_SUMMARY"
+  env:
+    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+```
+
+Output is always UTF-8, including on Windows runners.
+
+### Quality gates
+
+Reporting an issue does not fail the run. Gates do:
+
+```bash
+# Fail when a step is flaky or has regressed
+ci-time-tracker --github acme/web --fail-on flaky,regression
+
+# Fail when the median build takes longer than 10 minutes
+ci-time-tracker --github acme/web --max-duration 600
+```
+
+The report is written first, so a failing pipeline still shows why; the reasons
+are also printed to stderr. The median build duration counts each build once,
+leaving out the earlier attempts of re-runs.
+
+### Pull request comments (GitHub Action)
+
+Once CI finishes on a pull request, the action compares that run with the
+median of recent runs of the same workflow on the default branch and keeps a
+single comment on the pull request up to date:
+
+```markdown
+## ⏱️ CI time impact
+
+**CI** run 39102310449 on `9f8e7d6` took **8m 12s**: **🔺 +1m 30s (+22%)** compared
+with the median of the last 20 runs on `main`.
+
+| | `main` (median) | This run | Change |
+|---|---:|---:|---:|
+| Wall clock | 6m 42s | 8m 12s | 🔺 +1m 30s (+22%) |
+| Compute time | 36m 08s | 41m 10s | ≈ +5m 02s (+14%) |
+| Estimated cost | $0.29 | $0.33 | +$0.04 |
+
+### Steps that changed
+
+| Step | `main` (median) | This run | Change |
+|---|---:|---:|---:|
+| build / Run tests | 1m 52s | 2m 40s | 🔺 +48.0s (+43%) |
+
+### Failures
+
+- ❌ `integration / e2e`: ⚠️ known to be flaky (failed and passed on the same
+  commit in 2 of 15 commits). A re-run may pass.
+```
+
+Add it as a workflow of its own, triggered when CI completes:
+
+```yaml
+name: PR CI impact
+
+on:
+  workflow_run:
+    workflows: [CI]            # the name of your CI workflow
+    types: [completed]
+
+permissions:
+  actions: read
+  pull-requests: write
+
+jobs:
+  comment:
+    if: github.event.workflow_run.event == 'pull_request'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: 0xWhisp/ci-time-tracker@main   # pin a release tag once published
+```
+
+`workflow_run` runs the copy of the workflow on the default branch with the
+repository's own token, so it can comment on pull requests from forks without
+ever running their code. For the same reason, it takes effect once the
+workflow file is on the default branch.
+
+| Input | Default | |
+|---|---|---|
+| `baseline-branch` | default branch | Branch to compare with |
+| `baseline-runs` | `20` | Recent baseline runs to compare with |
+| `group-matrix` | `true` | Merge the legs of matrix jobs |
+| `regression-threshold` | `25` | Percentage (and at least 10s) for a step to be listed |
+| `comment` | `true` | Post or update the pull request comment |
+| `fail-on` | | `regression` fails when a step got slower; `flaky` when the run hit a known flaky step |
+| `run-id` | the triggering run | Workflow run to report on |
+| `github-token` | `github.token` | Needs `actions: read` and `pull-requests: write` |
+
+The same comparison is available from the command line:
+
+```bash
+ci-time-tracker --github acme/web --run-id 39102310449 --group-matrix
+ci-time-tracker --github acme/web --run-id 39102310449 --comment   # post it
+```
+
+### Exit codes
+
+| Code | Meaning |
+|------|---------|
+| 0 | Report written, gates passed |
+| 1 | File not found or unreadable, or nothing to analyze |
+| 2 | Invalid configuration, log, estimates or pricing file |
+| 3 | GitHub API error (rate limit, permissions, not found) |
+| 4 | A quality gate failed |
+
+## Cost Estimates
+
+Costs are estimated from the runner each job ran on, using GitHub's
+[list prices](https://docs.github.com/en/billing/reference/actions-runner-pricing)
+for **private repositories** as of October 2026, with each job rounded up to a
+whole minute. Public repositories pay nothing for standard GitHub-hosted
+runners; larger runners are billed for public repositories too.
+
+| Runner | USD per minute |
+|---|---:|
+| Linux 2-core (`ubuntu-*`) | 0.006 |
+| Linux 1-core (`ubuntu-slim`) | 0.002 |
+| Linux 2-core arm64 (`ubuntu-*-arm`) | 0.005 |
+| Windows 2-core, x64 or arm64 (`windows-*`) | 0.010 |
+| macOS 3/4-core (`macos-*`, `macos-*-intel`) | 0.062 |
+| macOS 12-core (`macos-*-large`) | 0.077 |
+| macOS 5-core M2 Pro (`macos-*-xlarge`) | 0.102 |
+
+- Versioned labels are recognized by family, so a new image such as
+  `ubuntu-26.04` or `macos-26` is priced without an update.
+- Larger runners with custom names are priced from the platform, architecture
+  and core count in the label (e.g. `ubuntu-24.04-8-cores`), using GitHub's
+  published rates. A core count GitHub does not publish stays unpriced.
+- Self-hosted runners cost `$0.00`.
+- Runners from other providers (Depot, Namespace, ...) and other custom names
+  have no known rate. They are listed under `unpriced_runners`, and the report
+  says how many jobs the estimate covers ("cost covers only 190 of 790 jobs"),
+  so a partial estimate is never shown as the whole cost.
+- Per-step costs are attributed proportionally and do not include the per-job
+  rounding, so they add up to slightly less than the job total.
+
+Prices change. Override any label, including the families above, with a JSON
+file of label to USD per minute:
+
+```bash
+ci-time-tracker --github acme/web --pricing pricing.json
+```
+
+```json
+{
+  "depot-ubuntu-24.04": 0.004,
+  "our-big-runner": 0.032
+}
+```
+
+## Matrix Builds
+
+GitHub names each leg of a matrix separately, so `Run tests` becomes
+`build (3.12, ubuntu-latest) / Run tests` and its siblings: one step per leg.
+`--group-matrix` merges them, which is usually what you want when asking where
+the time goes:
+
+```bash
+# Without grouping: "build (3.12, windows-latest) / Run tests  2m 59s  4.6%"
+# With grouping:    "build / Run tests                        45m 18s  69.2%"
+ci-time-tracker --github acme/web --group-matrix
+```
+
+Grouping mixes runners with different rates, so the cost of a merged step spans
+whatever machines its legs ran on.
+
+## Detection Criteria
+
+### Flaky Steps
+
+A step is **flaky** when it both failed and passed **on the same commit**: the
+code did not change, the outcome did. A failed run that passed when re-run is
+the clearest case, so the earlier attempts of re-run workflows are fetched too.
+
+- Outcomes are compared per workflow, step and commit, using each matrix leg's
+  own name: a leg that always fails on Windows is broken there, not flaky.
+- A high failure rate alone is **not** evidence. A step that fails because a
+  commit broke it, and passes once the fix lands, is doing its job.
+
+Logs without commit information (log files) fall back to the failure-rate
+heuristic (`0.05 < failure_rate < 0.95`), which cannot tell flakiness from real
+breakage. The report states which method was used (`flaky_detection`).
+
+### Regressions
+
+A step has **regressed** when its duration jumped and stayed up:
+
+1. Its history (one sample per build, oldest first) is split at the point that
+   best separates two stable segments.
+2. The later segment's median must be at least **25%** higher
+   (`--regression-threshold`) and at least **10 seconds** higher, so `1s -> 2s`
+   steps are not reported.
+3. The most recent 5 builds must still be that slow, so a spike that already
+   recovered is not reported.
+
+The report names the commit and build where the regression started. A step
+needs at least **10 builds** to be judged; `regression_checked_steps` says how
+many had enough history. With `--group-matrix`, the legs of a build are
+combined into one sample (their median).
+
+## Supported CI Providers
+
+| Provider | Config Format | Log Format |
+|----------|--------------|------------|
+| GitHub Actions | YAML | Text/JSON |
+| GitLab CI | YAML | Text/JSON |
+| CircleCI | YAML | Text/JSON |
+
+## Limitations
+
+- Log parsing relies on timestamp patterns; non-standard formats may not parse correctly
+- Flaky detection needs the same commit to have been built more than once
+- Regression detection needs at least 10 builds of a step
+- Estimated durations in config mode require user-provided estimates file
+
+## Development
+
+```bash
+# Install dev dependencies
+pip install -e ".[dev]"
+
+# Run tests
+pytest
+
+# Run with coverage
+pytest --cov=ci_time_tracker
+
+# Lint (rules are pinned in pyproject.toml)
+ruff check .
+```
+
+CI runs the suite on Linux and Windows for Python 3.10 to 3.14, then runs
+the tool on its own CI history: each run's summary page shows the resulting
+Markdown report.
+
+### Releasing
+
+Releases are published to PyPI by `.github/workflows/release.yml` when a
+version tag is pushed, using PyPI trusted publishing (no token in the repo).
+
+One-time setup on PyPI: add a trusted publisher for project `ci-time-tracker`
+with owner `0xWhisp`, repository `ci-time-tracker`, workflow `release.yml`
+and environment `pypi`, and create the `pypi` environment in the repository
+settings.
+
+Then, for each release:
+
+1. Bump `version` in `pyproject.toml` and `__version__` in
+   `ci_time_tracker/__init__.py` (a test checks they agree), and add the
+   release to [CHANGELOG.md](CHANGELOG.md).
+2. Tag and push: `git tag v0.2.0 && git push origin v0.2.0`.
+3. Create the GitHub release from the tag, with the changelog entry as its
+   notes, and tick "Publish this Action to the GitHub Marketplace".
+
+The workflow refuses to publish when the tag and the versions disagree.
+
+## License
+
+MIT License - see [LICENSE](LICENSE) for details.