3gpp-crawler 2026.11.10__tar.gz

3gpp_crawler-2026.11.10/.agents/FILES.md

@@ -0,0 +1,43 @@
+# AGENTS.md — FILES
+
+Single source of truth for paths, config keys, and naming conventions.
+Kept compact — agents hallucinate less when they know where definitions live.
+
+## Pattern
+
+- One file owns each class of definition (paths, config defaults, enums).
+- Import from that file. Never hard-code values in other modules.
+- Variables that address files get `_file` suffix; directories get `_dir`.
+
+## Project-specific sources of truth
+
+| What | Where | Key names |
+|------|-------|-----------|
+| Runtime paths | `src/tdoc_crawler/config/cache_manager.py` | `CacheManager`, `resolve_cache_manager()`, `DEFAULT_*` constants |
+| Config settings | `src/tdoc_crawler/config/settings.py` | `ThreeGPPConfig` (alias `TDocCrawlerConfig`), `PathConfig`, `HttpConfig`, `CredentialsConfig`, `CrawlConfig` |
+| Env vars | `.env.example` | `TDC_*`, `HTTP_CACHE_*`, `TDC_EOL_*`, `TDC_CRAWL_*` |
+| Domain constants | `src/tdoc_crawler/constants/` | `patterns.py` (regexes), `urls.py` |
+| Domain enums | `src/tdoc_crawler/models/base.py` | StrEnum: `TDocStatus`, `SourceKind`, `TagKind`, `OutputFormat`, `SortOrder` |
+| DB models | `src/tdoc_crawler/database/oxyde_models.py` | Oxyde models, one per table |
+| HTTP session | `src/tdoc_crawler/http_client/session.py` | `create_cached_session()`, `download_to_file()`, `PoolConfig` |
+| Docs config | `zensical.toml` (default builder; `mise run docs`) + `mkdocs.yml` (GitHub Pages workflow; `mise run docs:mkdocs`) | nav, theme, plugins - update BOTH navs when adding/removing pages |
+| Mise tasks/tools | `.config/mise/config.toml` + `conf.d/*.toml` | task definitions split by category; dev-feature tools/MCP/skills driven by `data/dev-features.csv` + `data/skills.csv` via `scripts/install_dev_features.py` (`mise run add-mcp-servers`, `mise run add-skills`) |
+| CI / release / docs / mirror | `.github/workflows/release.yml` (CalVer tag, GitHub release with LLM-drafted notes from `scripts/release_notes.py` falling back to `--generate-notes`, mirror push to forge), `.github/workflows/pypi.yml` (PyPI trusted publishing on published GitHub releases only, tag must match package version), `.github/workflows/docs.yml` (GitHub Pages) | Default branch `main` on GitHub; forge is a mirror only |
+| CLI commands | `src/tdoc_crawler/cli/` | Typer apps: `tdoc_app`, `spec_app`, `mgmt_app`, `config_app`, `tag_app`, `mcp_app` |
+| Credentials | `src/tdoc_crawler/credentials.py` | `set_credentials()`, EOL auth |
+
+## Naming conventions
+
+- **Archived plans** (`.agents/history/`, version-controlled):
+  `YYYY-MM-DD_<TYPE>_<summary>.md` — date = **completion date** of the
+  work, `TYPE` ∈ `SUMMARY` | `ANALYSIS` | `PLAN`. Example:
+  `2026-08-10_SUMMARY_feature1.md`.
+- **Active plans** (`.agents/plans/`, gitignored): no date prefix, but the
+  plan **must embed a date** (creation timestamp) in its content — e.g.
+  `Created: YYYY-MM-DD` line or a front-matter `date:` field. Filename
+  stays descriptive (`01-sub-feature.md`), order = implementation order.
+
+## Subdirectory
+
+If this file grows past its size budget, split detail into
+`.agents/files/<name>.md` — every file there MUST be listed above.

3gpp_crawler-2026.11.10/.agents/HISTORY.md

@@ -0,0 +1,80 @@
+# AGENTS.md — HISTORY
+
+Recorded decisions with git references. Read when relevant to current task.
+Acts as simple long-term memory for the project.
+
+Detailed change history: `.agents/history/` (committed summaries, chronological).
+
+## Format
+
+| Date | Decision | Rationale | Git ref |
+|------|----------|-----------|---------|
+| YYYY-MM-DD | [description] | [why] | [commit hash/tag] |
+
+## Record
+
+- High-level architectural decisions and breaking changes.
+- Dependency, tooling, and platform choices — including the alternative that lost.
+- False turns and rejected approaches, with the reason.
+- Completion of an implementation plan (link to its archive below).
+
+## Do not record
+
+- Routine commits, refactor logs, session activity, release notes.
+- Anything obvious from git, the code, or current AGENTS.md/state files.
+
+## Guidance
+
+- Record decisions that would be costly to rediscover.
+- Note false turns and why they were rejected.
+- Link to relevant commits.
+- Keep entries brief — enough to reconstruct reasoning.
+- If this file grows large, move older entries to `.agents/history/` and keep
+  a pointer here.
+
+## Archive a plan
+
+Archive when a plan's work lands — not at a later maintenance pass:
+
+1. Consolidate `.agents/plans/<feature>/` into a single file
+   `.agents/history/<YYYY-MM-DD>_<TYPE>_<summary>.md` (completion date;
+   naming schema in `.agents/FILES.md`).
+2. Delete the plan directory.
+3. Add an entry to `.agents/HISTORY.md` referencing the archive and its
+   commit.
+
+## Entries
+
+| Date | Decision | Rationale | Git ref |
+|------|----------|-----------|---------|
+| 2026-08-14 | Adopted agents-scaffold DOX hierarchy: lean root AGENTS.md (rules + index only), non-essential rules moved to `.agents/` (POLICIES, ONBOARDING, FILES), ⛔ No Patching guard | Always-injected AGENTS.md was bloated; every redundant byte dilutes every LLM call | (uncommitted) |
+| 2026-08-14 | `.agents/` DOX `.md` files now git-tracked (un-ignored); `/.agents/plans/` explicitly ignored | DOX files are durable records, plans are mutable working artifacts | (uncommitted) |
+| 2026-08-14 | Migrated tooling onto copier-uv-plus template: mise conf.d task layout, ty.toml + ty dev dep, zensical docs (replaces mkdocs), codespell/rumdl, template meta files, .copier-answers.yml adopted | Align tool set with template for future `copier update`; uv.lock stays untracked until stable v1 | (uncommitted) |
+| 2026-08-14 | zensical build panics on this machine (watcher.rs:299 "invariant", upstream #743 closed as not reproducible) — docs build deferred until fixed upstream or workaround found | Minimal repro at `%LOCALAPPDATA%\Temp\ztest` | (uncommitted) |
+| 2026-08-14 | forge.3gpp.org is GitLab EE mounted at `/rep` (relative URL root). glab works via: `GITLAB_TOKEN=` + `GITLAB_HOST=forge.3gpp.org` + `subfolder: rep` in glab config + running the glab binary directly (`mise which glab`) | mise re-injects GITLAB_TOKEN (gitlab.com token) into the env, which overrides the forge token and breaks API calls | (uncommitted) |
+| 2026-08-14 | Token split: `GITLAB_TOKEN` = gitlab.com (mise gitlab backend, glab-cli downloads); `MISE_GITLAB_ENTERPRISE_TOKEN` = forge.3gpp.org | Per mise docs token priority | (uncommitted) |
+| 2026-08-14 | Added `.gitlab-ci.yml`: on every push to default branch, compute next CalVer (vYYYY.MM.N from existing tags), create annotated tag + GitLab release with commit-log notes | Automatic versioning without CI runners was not possible (0 project runners); shared runners enabled on forge. `resource_group` serializes racing pushes; `RELEASE_TOKEN` variable overrides CI_JOB_TOKEN if 403 | (uncommitted) |
+| 2026-08-14 | **Replaced GitLab CI with pre-push hook** — forge has NO active shared runners (job stuck: "no active runners"). Other forge projects use group/project runners (e.g. `ivas-linux`); user-namespace project has none. `.gitlab-ci.yml` removed; `.githooks/pre-push` (installed via `git config core.hooksPath .githooks`) schedules `scripts/auto-release.sh` detached after the push lands; curl + `MISE_GITLAB_ENTERPRISE_TOKEN`, no glab/jq deps. First release v2026.08.1 created & verified | CI needs a registered runner to work; hook uses the already-verified forge API path | (uncommitted) |
+| 2026-08-14 | Moved detailed change history from `docs/history/` to `.agents/history/` (40 files, git-mv preserves history). Archive naming schema: `YYYY-MM-DD_<TYPE>_<summary>.md` (date = completion); active plans embed a date instead of a prefix. Schema lives in `.agents/FILES.md` | History is durable decision records — belongs under `.agents/`, not user docs; POLICIES previously suggested `.agents/history/<feature>/` subdirs, reconciled to the flat date-prefixed layout already in use | (uncommitted) |
+| 2026-08-14 | Enabled template dev-features (`include_mcp_tasks: True`): adopted `.config/mise/conf.d/{mcp,skills}.toml` + `data/{dev-features,skills}.csv` + `scripts/install_dev_features.py`; deleted hand-rolled `conf.d/ai.toml`; CSVs curated to project stack (grepai, teddi, cytoscnpy, beads; python-ultimate/ty/pydantic/etc. skills) | The CSV manifests are the template's designed extension point — the hand-written ai.toml was exactly what the skill forbids; project-specific rows (teddi, 3GPP skills) now live in the CSVs | (uncommitted) |
+| 2026-08-14 | Fixed `crawllogentry.created_at` NOT NULL violation on crawl start: oxyde 0.7.x `create()` uses `exclude_unset=True`, dropping `default_factory` fields from INSERTs; such fields have no SQL DEFAULT. `log_crawl_start` now sets `created_at` explicitly (other insert sites already set theirs; `_prepare_tdoc` normalizes TDocMetadata) | oxyde upgraded from 0.4.x floor to 0.7.1 by `uv sync`; documented in `database/AGENTS.md` Key Rules | (uncommitted) |
+| 2026-08-15 | auto-release.sh hardening: sed instead of python (bare `python` not on PATH in hook env), `LC_ALL=C.UTF-8`, description via `--data-urlencode description@file` with `cygpath -w` | git-bash → native curl.exe argv conversion mangles non-ASCII to `?`; `"\\n"` in double quotes stays literal so GitLab rendered it as text; MSYS `/tmp` paths unreadable for curl | (uncommitted) |
+| 2026-08-18 | Scope refocus: dropped workspace subsystem + all AI/extraction (docling, pymupdf, profiles, metrics). Added tags (polymorphic `TagEntry` model), `--pdf` flag on checkout, MCP server (fastmcp stdio, 14 grouped tools). Tags replace workspace membership as the lightweight grouping mechanism | Workspaces stalled; AI pipeline not needed here; MCP enables agent integration | 46c5b63 → f375f61 |
+| 2026-08-18 | Docs restructured for mkdocs (default builder; `mkdocs build --strict` clean): stale workspace/history/skills pages deleted, tags+mcp pages added, flags verified against real CLI, zensical.toml nav kept in sync (zensical still broken on Windows) | docs/ was a mix of historical paraphrase, broken nav (workspace.md), and stale flags; history belongs to `.agents/history/` | (this commit) |
+| 2026-08-19 | Added `reservation_date` + `uploaded_at` columns to TDocMetadata (issue #5), sourced from doc-list Excel `Reservation date` / `Uploaded` columns; added `--sort-by` (meeting_date/reservation_date/uploaded_at/tdoc_id) to query_tdocs CLI/MCP | TDoc numbers reflect reservation-slot order, not real upload order; callers need true submission time to sort. None values normalized to crawl time in `_prepare_tdoc` to avoid oxyde bulk "Missing column" | (uncommitted) |
+| 2026-08-20 | Fixed doubled `/Docs/Docs/` path in TDoc URLs (issue #6): `_construct_tdoc_url` now only appends `/Docs` when `files_url` (portal MtgDocURL) doesn't already end with it. Previously every crawled URL was a dead 403 link | `MtgDocURL` points at the meeting Docs dir; appending `/Docs/` again produced `.../Docs/Docs/<tdoc>.zip`. Checkout's `_resolve_corrected_url` stays as a safety net for legacy bad rows | (uncommitted) |
+| 2026-08-20 | Removed dead `.gitlab-ci.yml`. forge has no active runners on this project namespace (only group-level ones from `ivas-linux`), so every push triggered a pending→failed pipeline with empty trace and an email notification. Releases are already produced by `.githooks/pre-push` → `scripts/auto-release.sh` (CalVer tag + GitLab release + wheel upload), which is the entire purpose the CI served | Hook covers what CI would have done; CI cost was just the email noise | (uncommitted) |
+| 2026-08-24 | Rewrote `_run_migrations` in `database/base.py` to be model-driven: diff `extract_current_schema()` (the same source of truth that creates tables) against `PRAGMA table_info` introspected via `oxyde.execute_raw()`, and emit `ALTER TABLE ADD COLUMN` for every missing column. Replaces the hand-maintained hard-coded migration list that silently failed (the old code passed msgpack-encoded bytes from `raw` IR to a `for row in result` loop, throwing inside the catch-all `except` block). Fixes ``no such column: tdocmetadata.reservation_date`` on every query for any DB created before #5 landed | Migration code should be data-driven, not hand-maintained. Using `execute_raw` keeps the introspection on oxyde’s own connection (no stdlib sqlite3 side-channel). Type is inferred from `python_type`: `int` → INTEGER, everything else → TEXT (sqlite is dynamically typed anyway) | (this commit) |
+| 2026-08-24 | **Audit pass on `database/`** per user feedback: dropped `_ensure_tables_exist` / `_run_migrations` / `_PYTHON_TO_SQLITE_TYPE` / `_build_column_defs` / `_build_column_sql` (60+ lines of hand-rolled DDL) in favour of Oxyde's own `oxyde.db.schema.create_tables()` (dialect-aware, uses `migration_compute_diff` + `migration_to_sql` under the hood). Replaced the `table_to_model: dict[str, type[Model]]` with a typed `_clear_tables(models: Iterable[type])` that derives table names from `Model.get_table_name()`. **Killed migrations entirely** — on schema mismatch the DB raises `DatabaseError("schema-mismatch")` pointing at regeneration; no ALTER TABLE guessing. Live DB lost 31617 rows from the audit (test invocation of `clear_tdocs`) — user must re-crawl | Oxyde already knows how to create tables correctly; building our own DDL on top is over-engineering. The hardcoded `"tdocs"` / `"specs"` / `...` strings in `_clear_tables` were also drift magnets — one rename in `oxyde_models.py` and the clear silently no-op'd | (this commit) |
+| 2026-11-10 | Updated agents-scaffold to the 2026-10-17 template: new `.agents/TOOLS.md` state file (tool inventory moved from AGENTS.md + ONBOARDING), Hierarchy/Child Doc Shape sections, state-file subdirectory concept, plan-completion archive trigger | Framework update (INSTALL.md); project plan/archive naming kept deliberately (`.agents/plans/<feature>/` + `YYYY-MM-DD_<TYPE>_<summary>.md`) — upstream dated-dir naming NOT adopted | (this commit) |
+| 2026-11-10 | **Rebuilt beads tracking**: bd 0.49.6 server-mode Dolt store was broken (database `tdc` missing; stale `dolt sql-server` running against an empty `.beads/dolt/`). Re-initialized with bd 1.3.4 **embedded** Dolt (`.beads/embeddeddolt/`), re-imported all 45 issues from git-tracked `.beads/issues.jsonl`, removed legacy 0.49 state (`.beads/dolt/`, `dolt-server-*`, legacy config). Issues now sync to a NEW **private** repo `forge.3gpp.org/rep/reimes/3gpp-crawler-beads` (`refs/dolt/data`; `sync.remote` + `export.auto` in `.beads/config.yaml`; push auth `DOLT_REMOTE_USER=oauth2 DOLT_REMOTE_PASSWORD=$GITLAB_TOKEN`). Main code repo is publicly readable — never push beads data there | Upstream guidance: embedded is default for single-writer, Dolt remotes replaced the removed `sync-branch` workflow, "use private repos" (SECURITY.md); issues.jsonl is export-only, not a backup | (this commit) |
+| 2026-11-10 | **Moved primary hosting to GitHub** `github.com/jr2804/3gpp-crawler` (public). `forge.3gpp.org/rep/reimes/3gpp-crawler` is now a mirror: `.github/workflows/release.yml` pushes `main` + tags via project token `github-mirror-maintainer` (id 179, write_repository, Maintainer because forge `main` is protected for Maintainers-only push; expires 2027-11-10) stored as GitHub secret `FORGE_MIRROR_TOKEN`. Releases: CalVer tag + GitHub release created by workflow on each `main` push version change. PyPI publish gated by repo variable `PYPI_PUBLISH=true` (trusted publishing, env `pypi`). Docs: `.github/workflows/docs.yml` deploys MkDocs to GitHub Pages (`https://jr2804.github.io/3gpp-crawler/`). Removed `.githooks/pre-push`, `scripts/auto-release.sh`, `scripts/ci-release.sh` (forge-API release path). Local `branch.main.remote` = `github` (CalVer pre-commit reads tags there). Beads sync stays on forge private repo; beads data must never go to the public GitHub/forge code repos | Forge has no runners, so GitLab CI/Pages cannot run; GitHub Actions + Pages + PyPI trusted publishing need no self-hosted runner; forge kept for mirroring per user decision | (this commit) |
+| 2026-11-10 | **Removed PDF conversion**: deleted `src/tdoc_crawler/extraction/` (conversion + unused fetch helpers), `packages/convert-lo/`, `tests/convert_lo/`, `docs/convert-lo-usage.md`; `--pdf` removed from `tdoc checkout`, `spec checkout` and MCP `checkout_tdocs`/`checkout_specs` (breaking CLI/MCP change); pdf-remote-converter dependency and git source dropped; `OFFICE_FORMATS` moved to new `constants/` module (kept for planned archive classification) | Office->PDF existed only to feed knowledge/AI tooling, out of scope since the 2026-08-18 refocus; newer tools (parsecraft, docling, docling-rs) handle doc/docx themselves; pdf-remote-converter came from a non-PyPI git source, so PyPI installs failed on import | (this commit) |
+| 2026-11-10 | **Dependency bump + hishel 2 port**: hishel>=2.0.0 (hishel 2 dropped SyncSqliteStorage/SyncBaseStorage and the hishel.requests adapter for niquests). _NiquetsCacheAdapter now subclasses niquests HTTPAdapter and runs hishel's sync RFC 9111 state machine (hishel._sync_rfc.handle_request_with_rfc, private, no public sync equivalent) with SqliteStorage; cache key sha256(url). Cache file renamed http-cache.sqlite3 -> http-cache-v2.sqlite3 (hishel 2 refuses 1.x schema). Runtime lower bounds raised to the lock's resolved versions. LogLevel StrEnum (DEBUG..CRITICAL, uppercase values) now types all --verbosity options; numeric/lowercase levels are rejected by Typer | Blocker: hishel 2.0.0 was already installed by the lock, so the old SyncSqliteStorage import broke every CLI start; porting keeps RFC caching behaviour of hishel 1 default policy | (this commit) |
+| 2026-11-10 | **Cherry-pick migration copier-uv-plus 2026.8.5 → HEAD** (manual path-by-path, no `copier update`): installer upgraded (`enabled` CSV column, inline `tool_value` tables → `mise use --tool-option`, best-effort removal of disabled rows); CSVs reschema'd — repowise row deleted, beads/beads-mcp rows tool-stripped (tools live in `config.toml`), dormant rows added (dolt, desktop-commander, sequential-thinking, rust — all `enabled=0`); lint consolidated to upstream `ruff check src/ tests/ --fix --unsafe-fixes` (lint-fix/-unsafe tasks dropped); format paths `src/ tests/ scripts/`; **docs switched to zensical default via `--link-mode=copy` — this FIXES the 2026-08-14 Windows watcher panic** (build passes, 3.3s); mkdocs kept as `docs:mkdocs` for the Pages workflow; template `.config/pyreorder.toml` adopted (`runtime_setup` bucket); dead `[tool.undersort]` removed from pyproject; gen_credits no-`__file__` fallback; `.gitignore` +site//native binaries/agent dirs | Align with template HEAD for future template syncs; deviations kept deliberately: Typer CLI (real code, not the template's Cyclopts demo), static version in `__init__.py` (CalVer hook sed target), our `test_template_validation.py`, `uv.lock` untracked, `.copier-answers.yml` stays at 2026.8.5 (honest - cherry-pick, not render) | (uncommitted) |
+| 2026-11-10 | **Archive member tracking**: new `ArchiveMember` table (`database/archives.py`, `utils/archive.py`) records zip member listings of TDoc and spec archives, written on checkout and on explicit `tdoc files` / `spec files` (MCP `list_tdoc_files` / `list_spec_files`) requests; `--keep-zip` / MCP `keep_zip` keeps the TDoc zip (default: deleted, as before); `SpecificationDownload` (never written) removed | Users asked to browse attachments of tdocs/specs without a full download; storing listings avoids re-reading zips and keeps crawl free of eager archive reads | (uncommitted) |
+| 2026-11-10 | **Tag export/import + clear-db keeps tags + --output everywhere**: new `tag export` / `tag import` (natural keys kind/item_id/tag; JSON/YAML; `-f` file) in `database/tag_transfer.py`; `--clear-db` saves tags to `tags-backup.json` next to the DB and restores them after recreation (raw sqlite read works on stale schemas); `--output` on tag add/remove/list, `tdoc stats`, `tdoc checkout` (per-TDoc rows), `spec checkout`, `tdoc files`, `spec files` | Tags are user data that `--clear-db` used to destroy; natural keys survive PK changes and rebuilds; machine output must be pipeable (no progress bar on stdout) | (uncommitted) |
+| 2026-11-10 | **CLI structure cleanup — `tag` and `mcp` unmounted from the tdoc app**: `tdoc-crawler tag` / `tdoc-crawler mcp` (and `3gpp-crawler tdoc tag` / `tdoc-crawler tdoc mcp`) removed; both live once at the `3gpp-crawler` umbrella level. `tdoc-crawler` is now pure tdoc domain (crawl, crawl-meetings, query, query-meetings, open, files, stats, checkout, version + ct/cm/qt/qm aliases) | `mcp` was reachable under three names while missing under `spec` (infrastructure, not a tdoc concern); `tag` is cross-domain (TagKind includes SPEC) so nesting it under `tdoc` was conceptually wrong | (uncommitted) |
+| 2026-11-10 | **`database` command group (alias `db`) replaces scattered clear flags**: new `cli/database_app.py` mounted on `3gpp-crawler` with `clear-tdocs`, `clear-specs`, `clear-meetings` (new `MeetingDatabase.clear_meetings()`), `clear-all` (former `--clear-db`: delete DB file, save/restore tags via `tags-backup.json`, clear TDoc checkouts, `--yes` for scripts), `export-tags`, `import-tags` (moved from `tag export`/`tag import`); `--clear-db`/`--clear-tdocs`/`--clear-specs` removed from all `crawl`/`query` commands, `handle_clear_options()` deleted from `cli/_shared.py`; `schema-mismatch` message now names `3gpp-crawler database clear-all`; all six commands also exposed as MCP tools `database_*`, destructive ones gated on an explicit `confirm=true` parameter (no interactive prompt over MCP stdio); fixed latent circular import by importing `run_mcp` lazily in `cli/mcp_app.py` (`pytest tests/test_mcp.py` standalone used to fail) | Clear flags appeared on arbitrary crawl/query subcommands ("randomly"), and the schema-mismatch error pointed at a removed flag; destructive operations belong in one place that can grow (`database` = home for future DB tasks); agents may recover from schema-mismatch via MCP, but must opt in per call | (uncommitted) |
+| 2026-11-10 | **Config CLI: `config init PATH` positional, `config show -o FORMAT` (was `--format`)**: `config show` now prints the active config (was a stub); `config init` target is a positional argument, its `--format` stays; `ConfigFormat` StrEnum (toml/yaml/json) replaces the bare `Literal`; `tags-backup.json` is deleted after a successful `--clear-db` restore | Every data output takes `--output` and the `--output` option is no longer a mandatory option on `config init`; StrEnum rule covers config formats | (uncommitted) |
+| 2026-11-10 | **PyPI publish moved into the Release workflow**: `pypi` job added to `.github/workflows/release.yml` (downloads the `dist` artifact, trusted publishing via `id-token: write` + `pypi` environment), `pypi.yml` deleted; the PyPI trusted-publisher entry must point at workflow `release.yml` + environment `pypi` (registered as a pending publisher for project `3gpp-crawler`) | `pypi.yml` (`on: release: [published]`) never fired — releases are created with `GITHUB_TOKEN`, and GitHub suppresses events raised by `GITHUB_TOKEN`, so no other workflow triggers; consolidating the publish avoids the PAT that fixing `release.yml`'s token would require | (this commit) |

3gpp_crawler-2026.11.10/.agents/MAINTENANCE.md

@@ -0,0 +1,37 @@
+# AGENTS.md — MAINTENANCE
+
+How to keep `.agents/` files current.
+
+## Process
+
+1. When a rule changes, update the owning file — never another file.
+2. When adding a new state file or directory under `.agents/`, add it to the
+   index table in root `AGENTS.md` and to **File update triggers** below.
+3. After any `.agents/` change, check the index for stale entries.
+
+## Principles
+
+- **Single source of truth.** Each rule lives in exactly one file. The tier,
+  pointer, rationale, and size rules are in **DOX authoring**
+  (`.agents/POLICIES.md`) — consult it on every DOX change.
+- **AGENTS.md is an index.** It points to `.agents/` files, does not replace them.
+- **No logbooks.** State files describe current contracts; git owns history —
+  see `.agents/POLICIES.md`.
+- **State files stay lean.** When a file approaches its size budget, split
+  detail into the matching subdirectory (`.agents/files/`, `.agents/policies/`,
+  `.agents/tools/`, `.agents/maintenance/`) — every file there MUST be listed
+  in its state file.
+
+## File update triggers
+
+| File / Dir | Update when |
+| ------------ | ------------- |
+| `ONBOARDING.md` | Project structure, tooling, or entry points change |
+| `POLICIES.md` | Boundaries, priorities, or verification change |
+| `FILES.md` | Path constants, config keys, or naming conventions change |
+| `TOOLS.md` | Tools, MCP servers, skills, or CLIs added, removed, or changed |
+| `HISTORY.md` | Notable decision made or resolved |
+| `MAINTENANCE.md` | Maintenance procedures, triggers, or principles change |
+| `plans/` | New feature implementation starts |
+| `history/` | HISTORY.md overflow or completed plans archived (rename per naming schema in `FILES.md`) |
+| `files/`, `policies/`, `tools/`, `maintenance/` | State files split into subdirectories |

3gpp_crawler-2026.11.10/.agents/ONBOARDING.md

@@ -0,0 +1,70 @@
+# AGENTS.md — ONBOARDING
+
+Read this when starting a new session. After first read, only revisit when
+project structure or tooling changes significantly.
+
+## Project
+
+CLI tool for querying structured 3GPP document data (TDocs, Specs,
+Meetings). Full docs at `docs/` (see `docs/index.md`) and `README.md`.
+
+## Quick start
+
+```bash
+# Lint
+uv run ruff check src/ tests/
+
+# Test (all / single)
+uv run pytest -v
+uv run pytest tests/test_file.py -v
+
+# Coverage
+uv run pytest --cov=src --cov-report=term-missing
+
+# Add dependency
+uv add <package>
+```
+
+## Entry points (read these first)
+
+| File | Why |
+|------|-----|
+| `AGENTS.md` | Root rail — rules + `.agents/` index |
+| `.agents/POLICIES.md` | Boundaries, priorities, verification |
+| `.agents/FILES.md` | Source-of-truth locations |
+| `docs/development.md` | Setup, workflow, architecture |
+| `docs/index.md` | Documentation index |
+
+## Where to dig deeper
+
+- `docs/` — user-facing documentation (crawl, query, tags, config)
+- `.agents/TOOLS.md` — tools, MCP servers, skills, CLIs in use
+- `.agents/history/` — detailed change history
+- `.agents/HISTORY.md` — past decisions and rationale
+- Source-tree AGENTS.md files — local contracts per area
+- `.agents/skills/3gpp-*/SKILL.md` — 3GPP domain context
+
+## Terminology
+
+| Term | Means |
+|------|-------|
+| TDoc | 3GPP Temporary Document (e.g., `S4-250638`) |
+| Spec | 3GPP Technical Specification (TS/TR, e.g., `TS 26.444`) |
+| WG | Working Group (e.g., S4, RAN1, CT3) |
+| TSG | Technical Specification Group (SA, RAN, CT) |
+| Portal | 3GPP EOL authenticated portal |
+
+## Architecture
+
+- `tdocs/`, `specs/`, `meetings/` — domain packages; operations via
+  explicit submodule imports (no re-exports, circular import prevention)
+- `database/` — Oxyde ORM, one model per table in `oxyde_models.py`
+- `cli/` — thin Typer/Rich wrappers only
+- `config/` — `ThreeGPPConfig` (settings) + `CacheManager` (runtime paths)
+- `http_client/` — cached sessions (`create_cached_session()`)
+
+## Available tools
+
+Development tools, MCP servers, skills, and CLIs are inventoried in the
+state file `.agents/TOOLS.md` — read it before assuming a tool, update it
+when tooling changes.

3gpp_crawler-2026.11.10/.agents/POLICIES.md

@@ -0,0 +1,228 @@
+# AGENTS.md — POLICIES
+
+Always applicable. Boundaries, priorities, verification, checklist.
+
+## Priorities
+
+1. Correctness
+2. Evidence
+3. Safety
+4. Minimal changes
+5. Consistency
+6. Performance
+
+## Boundaries
+
+- NEVER fabricate paths, commits, APIs, config keys, env vars, test results,
+  or benchmark numbers. State gaps explicitly.
+- NEVER guess at command names, flags, or paths. Read source or run `--help`.
+- NEVER game verification by weakening assertions, narrowing scope, reducing
+  coverage, or skipping checks.
+- NEVER add secrets, API keys, or tokens to files or logs. If encountered,
+  note the location and stop.
+- NEVER run destructive commands (`rm -rf`, `git reset --hard`,
+  `git push --force`) without explicit confirmation.
+- NEVER delete or move files without explicit instruction.
+- NEVER create temp files in project root. Use a dedicated temp dir.
+- Be direct. Avoid flattery, filler, and agreeing with incorrect premises.
+
+## Change constraints
+
+- Minimal, surgical edits. Preserve existing style.
+- No new dependencies without explicit instruction. Prefer existing deps.
+- No unrelated refactoring while fixing a bug. Note adjacent issues separately.
+- Propagate failures with existing error patterns; never swallow silently.
+- Check injection, path traversal, unvalidated input, auth bypass, and
+  secret leakage risks on every change.
+
+## Uncertainty
+
+- Ask before acting when intent is materially ambiguous, or when a choice
+  changes behavior, API/UX, naming, persistence, auth, dependencies, config,
+  or compatibility. Prefer one targeted question; bundle only independently
+  answerable ones. Proceed when ambiguity is low-risk and conventions
+  decide — state the assumption briefly.
+
+## Evidence
+
+Gather evidence proportional to risk: trivial edits → target file + adjacent
+context; behavioral/API/dependency/infra changes → trace execution path,
+call sites, constraints, regression surface first. Check local code, imports,
+config, types, tests before assuming behavior. If local dependency code is
+unreadable, check upstream docs or source. Prefer external verification
+(fresh test) over self-review; state uncertainty when unconfirmed.
+
+## Workflow
+
+1. Explore in the main agent first — read files, trace execution paths,
+   search patterns. Never delegate before you have seen the data.
+2. Scan skills for direct and adjacent matches before choosing the path.
+3. Single-track or dependent steps stay in the main agent; 2+ independent
+   tracks → launch all subagents in the same response. Use 2+ or none,
+   NEVER exactly 1.
+4. Implement the smallest correct change; run the narrowest relevant check.
+
+## Subagents
+
+Subagent calls block the main agent — launch all in one batch. Each prompt
+requires a concrete return format (specific answer, list, or summary),
+never "report findings". Tracks must not depend on each other's results.
+Don't hand off data already in main-agent context for formatting or
+transformation.
+
+## Testing
+
+- Preserve existing tests; update them when behavior changes. Never silently
+  change tested behavior.
+- Scope validation proportionally: docs → readback; types → typecheck;
+  runtime/UI → targeted test, lint, or build.
+- State pre-existing failures without attributing them to your work. One
+  targeted fix if a post-change failure cause is clear, else stop and report.
+- Non-trivial logic (branch, loop, parser, money/security path) leaves ONE
+  runnable check: assert-based `demo()`/`__main__` or one small `test_*.py`.
+  Trivial one-liners need no test.
+
+## Git & PRs
+
+Commit only when explicitly requested; message states the change and why.
+PRs small, one concern. No force-push to main/master; no `--no-verify` or
+`--no-gpg-sign`.
+
+## Code quality
+
+Universal defaults. Project-specific standards live in child AGENTS.md.
+
+- **SRP** — one reason to change per module/function.
+- **DRY** — check before adding. Extract on third occurrence (Rule of Three).
+- **Redundant code** — remove dead branches, unreachable conditions, unused
+  params/imports before finishing.
+- **Duplication vs. abstraction** — prefer duplication over wrong premature
+  abstraction.
+- No new code-quality tooling by default — opt in per project.
+
+## Completion checklist
+
+- Change solves the stated problem.
+- Relevant validation ran (or gaps stated).
+- No known unintended side effects; no secrets added or exposed.
+
+## Verification
+
+```bash
+# Lint
+uv run ruff check src/ tests/
+
+# Test (all / single)
+uv run pytest -v
+uv run pytest tests/test_file.py -v
+
+# Coverage
+uv run pytest --cov=src --cov-report=term-missing
+
+# Add dependency
+uv add <package>
+```
+
+Note: `ty` is not installed — do not run `uv run ty check` unless added.
+
+## Response format
+
+Concise and specific. No filler, intros, or restated requirements.
+Answer direct questions directly.
+
+For review/debugging/analysis: findings with references, conclusion,
+approach. Mention caveats and unverified risks.
+
+## DOX authoring (keep AGENTS.md lean)
+
+AGENTS.md and `.agents/` files share limited context. Bloat and duplication
+are the failure mode.
+
+### Where a rule lives
+
+A rule goes in the **highest (most general) tier that fully applies**:
+
+- Applies to **all code in the project** → `.agents/POLICIES.md` (default).
+- Meaningful **only inside one subtree** → that subtree's `AGENTS.md`.
+- Just helps navigation → the parent's Child DOX Index, nothing else.
+
+### Reference, don't restate
+
+A rule appears in **exactly one file**. Other files use a pointer line:
+`<topic> — see <file>`. Never copy the rule, rationale, or example.
+
+### Rule first, rationale second, never third
+
+A DOX line is the rule. If a rationale is costly to rediscover, add one
+short clause. No multi-paragraph explanations or code pairs — those belong
+in a skill or `docs/`, linked once.
+
+### Content rules
+
+- **No tree views.** Generate on demand with `rg --files | tree-cli --fromfile`.
+- **No history in AGENTS.md files.** Git log has routine history. Record only
+  decisions costly to rediscover — in `.agents/HISTORY.md` (or
+  `.agents/history/` for overflow archives).
+- **Lean state files, optional subdirectories.** State files stay brief entry
+  points (index-style, not sprawling documents). A state file that grows past
+  its size budget (≤ **120 lines**) must be split into
+  `.agents/<topic>/<name>.md` subdirectories — not stuffed longer. Every file
+  in an optional subdirectory **MUST** be mentioned in the state file that
+  owns it. See **State file subdirectories** below for the naming map.
+- **No TODO lists.** Use issue trackers or beads.
+- Prefer direct bullets with explicit names.
+- Trim obvious statements, repeated rules, and detail that no longer applies.
+
+## Implementation Plans
+
+- Plans, refactoring plans, and design docs go in `.agents/plans/<feature>/`
+  — NOT under version control (gitignored).
+- Name descriptively with phase-number prefixes so file order matches
+  implementation order, e.g. `01-sub-feature.md`, `02-next-sub-feature.md`.
+- NEVER commit plans to git. They are mutable working artifacts.
+- The archive trigger is **plan completion**: archive the moment the work
+  lands — don't wait for the next maintenance pass (see **Documentation**).
+- Shipped docs (user-facing guides, API references) go in `docs/` and
+  ARE committed.
+
+## State file subdirectories
+
+Each state file has a matching optional subdirectory for growth — the file
+stays a lean index and the subdirectory carries the detail:
+
+| State file | Optional subdirectory |
+| ---------- | --------------------- |
+| `FILES.md` | `.agents/files/` |
+| `POLICIES.md` | `.agents/policies/` |
+| `TOOLS.md` | `.agents/tools/` |
+| `MAINTENANCE.md` | `.agents/maintenance/` |
+
+- Only create a subdirectory when its state file approaches its size budget.
+- Every file in a subdirectory MUST be listed in its state file (name + one
+  line of what it covers).
+- `ONBOARDING.md` and `HISTORY.md` have no subdirectories: entry points stay
+  a single file, and history growth goes to `.agents/history/`.
+
+## Documentation
+
+- When `.agents/HISTORY.md` grows too large, archive older entries to
+  `.agents/history/`. Reference them from `.agents/HISTORY.md`.
+- Completed plans (`.agents/plans/<feature>/`) are archived to
+  `.agents/history/` for long-term reference — rename with completion-date
+  prefix per the naming schema in `.agents/FILES.md`.
+- `.agents/history/` IS version-controlled (durable decision records).
+- AGENTS.md and state files describe the project's current contracts — never
+  append session notes, iteration logs, or changelogs; delete stale text
+  instead. `.agents/HISTORY.md` is the only permitted exception and holds
+  high-level decisions only.
+
+## Size budget
+
+- `.agents/` files: ≤ **120 lines** (always-injected context).
+- Subtree `AGENTS.md`: ≤ **250 lines** (loaded cumulatively with parents).
+
+Exceeding the budget signals restating instead of pointing, or hoarding
+rationale. Cut first; split the subtree only as a last resort.
+
+When a state file exceeds its budget, split detail into its matching
+subdirectory — see **State file subdirectories** above.

3gpp_crawler-2026.11.10/.agents/TOOLS.md

@@ -0,0 +1,52 @@
+# AGENTS.md — TOOLS
+
+State file: inventory of tools used in development — CLIs, MCP servers,
+skills, runtimes, and project scripts. Demand-loaded like all state files;
+keep it current (see `.agents/MAINTENANCE.md`).
+
+## Rules
+
+- Use an existing tool before adding a new one. No new tools, MCP servers,
+  skills, or dependencies without explicit instruction (`.agents/POLICIES.md`).
+- Each tool appears exactly once, in its primary table.
+- Tables describe current usage — delete rows when tools leave; this is not a
+  historical log.
+
+## Subdirectory
+
+If this file grows past its size budget, split detail into
+`.agents/tools/<name>.md` — every file there MUST be listed above.
+
+## CLI tools & runtimes
+
+| Tool | When | Purpose |
+| ---- | ---- | ------- |
+| `uv` | Python deps & scripts | Package manager: `uv run`, `uv add` |
+| `mise` | Running tasks | Task runner; definitions in `.config/mise/conf.d/*.toml` |
+| `bd` / beads | Issue tracking | Task lifecycle, dependencies, session persistence; embedded Dolt store synced to private repo `3gpp-crawler-beads` (push: `DOLT_REMOTE_USER=oauth2 DOLT_REMOTE_PASSWORD=$GITLAB_TOKEN bd dolt push`) |
+| `glab` | GitLab (forge.3gpp.org) | Forge mirror admin (tokens, mirror checks); primary repo is GitHub |
+| `gh` | GitHub (github.com/jr2804/3gpp-crawler) | Primary repo, releases, Pages, secrets (`FORGE_MIRROR_TOKEN`) |
+
+## MCP servers
+
+| Server | When | Purpose |
+| ------ | ---- | ------- |
+| `grepai` | Finding code | Semantic search by meaning, not text; call graphs |
+| `teddi-mcp` | Terms/definitions | ETSI TEDDI queries, technical body codes |
+
+## Skills
+
+| Skill | When | Purpose |
+| ----- | ---- | ------- |
+| `3gpp-*` | Any 3GPP work | Domain context: tdocs, specs, meetings, releases, WGs, portal auth |
+| `python-ultimate` / `pydantic` / `ty-skills` | Python work | Coding standards, pydantic, type checking |
+| `ponytail` / `caveman` | Code / prose | Minimal code / terse communication modes |
+
+## Project scripts & tasks
+
+| Command | Purpose |
+| ------- | ------- |
+| `mise run lint` / `typecheck` / `test` | Ruff / ty / pytest+coverage |
+| `mise run format-md` | rumdl markdown lint+fmt |
+| `mise run docs` | zensical build (`docs:mkdocs` = mkdocs for the Pages workflow) |
+| `mise run add-mcp-servers` / `add-skills` | Install from `data/dev-features.csv` / `data/skills.csv` |

3gpp_crawler-2026.11.10/.agents/history/2025-10-20_SUMMARY_01_targeted_fetch_documentation.md

@@ -0,0 +1,5 @@
+# Summary – 2025-10-20
+
+- Added `docs/QUICK_REFERENCE.md` as the authoritative CLI reference.
+- Documented targeted TDoc fetch behaviour triggered by `query` and `open` commands.
+- Described the end-to-end `open` workflow, including download, extraction, and launch details.