tele-mess-core 0.3.0__tar.gz

tele_mess_core-0.3.0/LICENSE

@@ -0,0 +1,186 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction, and
+distribution as defined by Sections 1 through 9 of this document.
+
+"Licensor" shall mean the copyright owner or entity authorized by the
+copyright owner that is granting the License.
+
+"Legal Entity" shall mean the union of the acting entity and all other
+entities that control, are controlled by, or are under common control with
+that entity. For the purposes of this definition, "control" means (i) the
+power, direct or indirect, to cause the direction or management of such
+entity, whether by contract or otherwise, or (ii) ownership of fifty percent
+(50%) or more of the outstanding shares, or (iii) beneficial ownership of
+such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity exercising
+permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications,
+including but not limited to software source code, documentation source, and
+configuration files.
+
+"Object" form shall mean any form resulting from mechanical transformation
+or translation of a Source form, including but not limited to compiled object
+code, generated documentation, and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or Object form,
+made available under the License, as indicated by a copyright notice that is
+included in or attached to the work (an example is provided in the Appendix
+below).
+
+"Derivative Works" shall mean any work, whether in Source or Object form,
+that is based on (or derived from) the Work and for which the editorial
+revisions, annotations, elaborations, or other modifications represent, as a
+whole, an original work of authorship. For the purposes of this License,
+Derivative Works shall not include works that remain separable from, or
+merely link (or bind by name) to the interfaces of, the Work and Derivative
+Works thereof.
+
+"Contribution" shall mean any work of authorship, including the original
+version of the Work and any modifications or additions to that Work or
+Derivative Works thereof, that is intentionally submitted to Licensor for
+inclusion in the Work by the copyright owner or by an individual or Legal
+Entity authorized to submit on behalf of the copyright owner. For the purposes
+of this definition, "submitted" means any form of electronic, verbal, or
+written communication sent to the Licensor or its representatives, including
+but not limited to communication on electronic mailing lists, source code
+control systems, and issue tracking systems that are managed by, or on behalf
+of, the Licensor for the purpose of discussing and improving the Work, but
+excluding communication that is conspicuously marked or otherwise designated
+in writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity on
+behalf of whom a Contribution has been received by Licensor and subsequently
+incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of this
+License, each Contributor hereby grants to You a perpetual, worldwide,
+non-exclusive, no-charge, royalty-free, irrevocable copyright license to
+reproduce, prepare Derivative Works of, publicly display, publicly perform,
+sublicense, and distribute the Work and such Derivative Works in Source or
+Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of this
+License, each Contributor hereby grants to You a perpetual, worldwide,
+non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this
+section) patent license to make, have made, use, offer to sell, sell, import,
+and otherwise transfer the Work, where such license applies only to those
+patent claims licensable by such Contributor that are necessarily infringed by
+their Contribution(s) alone or by combination of their Contribution(s) with
+the Work to which such Contribution(s) was submitted. If You institute patent
+litigation against any entity (including a cross-claim or counterclaim in a
+lawsuit) alleging that the Work or a Contribution incorporated within the Work
+constitutes direct or contributory patent infringement, then any patent
+licenses granted to You under this License for that Work shall terminate as of
+the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the Work or
+Derivative Works thereof in any medium, with or without modifications, and in
+Source or Object form, provided that You meet the following conditions:
+
+(a) You must give any other recipients of the Work or Derivative Works a copy
+of this License; and
+
+(b) You must cause any modified files to carry prominent notices stating that
+You changed the files; and
+
+(c) You must retain, in the Source form of any Derivative Works that You
+distribute, all copyright, patent, trademark, and attribution notices from the
+Source form of the Work, excluding those notices that do not pertain to any
+part of the Derivative Works; and
+
+(d) If the Work includes a "NOTICE" text file as part of its distribution,
+then any Derivative Works that You distribute must include a readable copy of
+the attribution notices contained within such NOTICE file, excluding those
+notices that do not pertain to any part of the Derivative Works, in at least
+one of the following places: within a NOTICE text file distributed as part of
+the Derivative Works; within the Source form or documentation, if provided
+along with the Derivative Works; or, within a display generated by the
+Derivative Works, if and wherever such third-party notices normally appear.
+The contents of the NOTICE file are for informational purposes only and do not
+modify the License. You may add Your own attribution notices within Derivative
+Works that You distribute, alongside or as an addendum to the NOTICE text from
+the Work, provided that such additional attribution notices cannot be construed
+as modifying the License.
+
+You may add Your own copyright statement to Your modifications and may provide
+additional or different license terms and conditions for use, reproduction, or
+distribution of Your modifications, or for any such Derivative Works as a
+whole, provided Your use, reproduction, and distribution of the Work otherwise
+complies with the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise, any
+Contribution intentionally submitted for inclusion in the Work by You to the
+Licensor shall be under the terms and conditions of this License, without any
+additional terms or conditions. Notwithstanding the above, nothing herein
+shall supersede or modify the terms of any separate license agreement you may
+have executed with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade names,
+trademarks, service marks, or product names of the Licensor, except as
+required for reasonable and customary use in describing the origin of the Work
+and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or agreed to in
+writing, Licensor provides the Work (and each Contributor provides its
+Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied, including, without limitation, any warranties
+or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+PARTICULAR PURPOSE. You are solely responsible for determining the
+appropriateness of using or redistributing the Work and assume any risks
+associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory, whether in
+tort (including negligence), contract, or otherwise, unless required by
+applicable law (such as deliberate and grossly negligent acts) or agreed to in
+writing, shall any Contributor be liable to You for damages, including any
+direct, indirect, special, incidental, or consequential damages of any
+character arising as a result of this License or out of the use or inability
+to use the Work (including but not limited to damages for loss of goodwill,
+work stoppage, computer failure or malfunction, or any and all other
+commercial damages or losses), even if such Contributor has been advised of
+the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing the Work
+or Derivative Works thereof, You may choose to offer, and charge a fee for,
+acceptance of support, warranty, indemnity, or other liability obligations
+and/or rights consistent with this License. However, in accepting such
+obligations, You may act only on Your own behalf and on Your sole
+responsibility, not on behalf of any other Contributor, and only if You agree
+to indemnify, defend, and hold each Contributor harmless for any liability
+incurred by, or claims asserted against, such Contributor by reason of your
+accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+APPENDIX: How to apply the Apache License to your work.
+
+To apply the Apache License to your work, attach the following boilerplate
+notice, with the fields enclosed by brackets "[]" replaced with your own
+identifying information. (Don't include the brackets!) The text should be
+enclosed in the appropriate comment syntax for the file format. We also
+recommend that a file or class name and description of purpose be included on
+the same "printed page" as the copyright notice for easier identification
+within third-party archives.
+
+Copyright 2026 dreaifekks
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

tele_mess_core-0.3.0/PKG-INFO

@@ -0,0 +1,365 @@
+Metadata-Version: 2.4
+Name: tele-mess-core
+Version: 0.3.0
+Summary: Telegram message archive core with SQLite storage and sync API
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/dreaifekks/tele-mess-core
+Project-URL: Repository, https://github.com/dreaifekks/tele-mess-core
+Project-URL: Issues, https://github.com/dreaifekks/tele-mess-core/issues
+Project-URL: Changelog, https://github.com/dreaifekks/tele-mess-core/blob/master/CHANGELOG.md
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: Telethon<2,>=1.38
+Requires-Dist: PyYAML<7,>=6
+Dynamic: license-file
+
+# tele-mess-core
+
+`tele-mess-core` is a single-user, multi-Telegram-account archive core for
+future Mac and web clients.
+
+It stores Telegram messages in SQLite and exposes sync plus management endpoints
+for remote clients. The core is meant to run as a long-running local database and
+control plane: clients manage account authentication, origin discovery, backup
+selection, capture policy, topics, and participant metadata through the core API.
+
+It intentionally does not forward messages to backup Telegram groups. Daily
+package generation and local Codex-backed daily summaries run as local jobs
+against the archive. See
+[product direction](https://github.com/dreaifekks/tele-mess-core/blob/master/docs/product-direction.md)
+for the management interface direction and
+[daily packaging](https://github.com/dreaifekks/tele-mess-core/blob/master/docs/daily-packaging.md)
+for daily package and AI analysis details.
+
+## Current Scope
+
+- Telegram ingestion with Telethon.
+- Multiple Telegram accounts feeding one archive.
+- One supervised, long-lived Telethon client per account, shared by ingestion,
+  auth, discovery, participant refresh, and summary delivery.
+- SQLite archive for chats, users, messages, reactions, and event cursors.
+- Cursor-based HTTP sync API for LAN or Tailscale use.
+- Token-protected management API for account state, origins, backup policies,
+  topics, participant metadata, and capture cursors.
+- Built-in web console for the same management surface at `GET /console`.
+- Policy-aware ingestion with bounded history backfill and reconnect catch-up.
+- Live origin discovery and participant refresh endpoints for authenticated
+  Telegram sessions.
+- Runtime operation events for Telegram auth/discovery/media-download failures.
+- Server daemon mode for an always-on Linux deployment.
+- macOS-oriented local CLI mode with durable jobs and no HTTP listener by
+  default.
+- Daily package generation by origin, tag group, timezone, and local date.
+- Local Codex-backed daily analysis with important-origin full-context reports,
+  all-origin structured message points, and point-based daily digests.
+- Durable daily package-and-summary jobs with deduplication, cancellation,
+  restart recovery, leases, and a retryable Telegram delivery outbox.
+- System-managed daily package and summary scheduling through user-level systemd
+  timer files.
+- Optional raw Telegram JSON retention cleanup for keeping the SQLite archive
+  compact while preserving structured message rows.
+
+## Run from PyPI with uvx
+
+[`uvx`](https://docs.astral.sh/uv/guides/tools/) installs the published package
+and its Python dependencies into an isolated cache, then runs the
+`tele-mess-core` command. It does not clone the repository into the current
+directory, and runtime state remains in the selected workspace rather than the
+tool environment.
+
+Once a release is available on PyPI and the Mac workspace contains a valid
+`config.yml`, the local core is one command:
+
+```bash
+uvx tele-mess-core run-local
+```
+
+Use the same stable workspace across invocations, or select one explicitly:
+
+```bash
+uvx tele-mess-core run-local \
+  --workspace "$HOME/Library/Application Support/tele-mess-core-personal"
+```
+
+For a reproducible launcher, pin the distribution while keeping the executable
+name explicit:
+
+```bash
+uvx --from "tele-mess-core==X.Y.Z" tele-mess-core run-local
+```
+
+Maintainers can find the tag and Trusted Publishing process in the
+[release guide](https://github.com/dreaifekks/tele-mess-core/blob/master/docs/releasing.md).
+
+A fresh machine still needs Telegram API credentials and a local configuration
+before `run-local` can ingest messages. In the macOS product, `mess-end` owns
+that first-run UI and drives the core management/auth API for code/2FA login,
+origin discovery, and capture-policy management; these are intentionally not
+duplicated as interactive core CLI setup. A standalone operator may enable
+`run-local --web` temporarily as a fallback. The normal local runtime keeps HTTP
+disabled.
+
+## Source Checkout
+
+```bash
+python3 -m venv .venv
+. .venv/bin/activate
+pip install -e .
+cp config.example.yml config.yml
+tele-mess-core init-db --config config.yml
+tele-mess-core smoke-telegram --config config.yml
+tele-mess-core run-server --config config.yml
+```
+
+If no authorized Telegram session exists, use the built-in console while
+`run-server` is active to request a login code and submit code/2FA credentials.
+See the
+[server-mode guide](https://github.com/dreaifekks/tele-mess-core/blob/master/docs/server-mode.md)
+for the always-on deployment shape and client sync contract. See the
+[daily-packaging guide](https://github.com/dreaifekks/tele-mess-core/blob/master/docs/daily-packaging.md)
+for the daily packaging, scheduling, and staged AI analysis workflow.
+
+## macOS Local Mode
+
+`run-local` starts Telegram ingestion plus the durable daily worker without
+starting the HTTP API or web console. On macOS its default workspace is
+`~/Library/Application Support/tele-mess-core`, so it does not depend on the
+Terminal or launcher current directory.
+
+```bash
+mkdir -p "$HOME/Library/Application Support/tele-mess-core"
+cp config.example.yml "$HOME/Library/Application Support/tele-mess-core/config.yml"
+tele-mess-core paths
+tele-mess-core run-local
+```
+
+Use `--workspace PATH` (alias `--work-dir`),
+`TELE_MESS_CORE_WORKSPACE`, or `TELE_MESS_CORE_HOME` to select another stable
+instance root. `TELE_MESS_CORE_CONFIG` or an explicit `--config` selects a
+specific config. `tele-mess-core paths` prints the resolved non-secret paths
+without opening the database.
+
+HTTP remains opt-in in local mode:
+
+```bash
+tele-mess-core run-local --web
+```
+
+This enables the existing API and `/console`; it never opens a browser
+automatically. See the
+[local-mode guide](https://github.com/dreaifekks/tele-mess-core/blob/master/docs/local-mode.md)
+for precedence, path semantics, first-login limitations, and configuration
+examples.
+
+Use `telegram.accounts[]` for multi-account auth/runtime configuration. Message
+capture sources are managed in SQLite through origin discovery plus backup
+policies; `telegram.chats` in config is no longer used.
+
+## Raw JSON Cleanup
+
+Message rows keep structured fields plus a raw Telethon JSON payload for recent
+forensics. The raw payload can be cleared after a retention window without
+removing message text, timestamps, senders, search data, or sync cursors.
+
+```bash
+tele-mess-core cleanup-raw-json --config config.yml --retention-days 7
+tele-mess-core cleanup-raw-json --config config.yml --retention-days 7 --dry-run
+tele-mess-core raw-json-cleanup-schedule --config config.yml install --activate-systemd
+```
+
+The cleanup timer defaults to `OnCalendar=weekly` and reads
+`storage.raw_json_retention_days`, which defaults to `7`. Add `--vacuum` only
+when you want the SQLite file to shrink immediately; without it, SQLite reuses
+the freed pages for later messages.
+
+## Sync API
+
+- `GET /healthz`
+- `GET /sync/state`
+- `GET /sync/events?after=0&limit=500`
+- `GET /sync/messages?after=0&limit=500`
+- `GET /sync/accounts`
+- `GET /sync/chats`
+- `GET /sync/search?q=term`
+- `GET /sync/media-files?account_id=main`
+
+## Management API
+
+- `GET /manage/capabilities`
+- `GET /manage/accounts`
+- `POST /manage/accounts`
+- `POST` or `PATCH /manage/accounts/auth`
+- `POST /manage/accounts/auth/status`
+- `POST /manage/accounts/auth/request-code`
+- `POST /manage/accounts/auth/submit-code`
+- `GET /manage/origins?account_id=main`
+- `POST /manage/origins`
+- `GET /manage/backup-policies?account_id=main`
+- `POST` or `PATCH /manage/backup-policies`
+- `GET /manage/participants?account_id=main&origin_id=-100123`
+- `POST /manage/participants`
+- `GET /manage/capture-cursors?account_id=main`
+- `GET /manage/operation-events?account_id=main&status=failed`
+- `GET` or `PATCH /manage/daily-package-schedule`
+- `GET` or `PATCH /manage/daily-summary-delivery`
+- `POST /manage/daily-packages`
+- `GET /manage/daily-package-runs`
+- `POST /manage/daily-summaries`
+- `POST` or `GET /manage/daily-summary-jobs`
+- `PATCH /manage/daily-summary-jobs/cancel`
+- `GET /manage/daily-summary-runs`
+- `GET /manage/daily-summary-records`
+- `GET /manage/daily-summary-records/item`
+- `GET /manage/daily-message-points`
+- `GET /manage/daily-message-points/item`
+- `POST /manage/discover-origins`
+- `POST /manage/participants/refresh`
+- `GET /console`
+
+The authoritative API reference is generated from
+`src/tele_mess_core/server/contracts.py`:
+
+- `docs/api.md` for human-readable endpoint docs.
+- `docs/openapi.json` for tools.
+- `docs/api-agent.md` for short agent lookup.
+- `GET /manage/api-manifest` for the runtime contract version/hash and route
+  registry.
+- `GET /openapi.json` and `GET /docs/api.md` for runtime docs served by the
+  core process.
+
+Regenerate and verify these files with:
+
+```bash
+tele-mess-core generate-api-docs
+tele-mess-core generate-api-docs --check
+```
+
+`GET /console` serves the built-in management console. The page can be opened in
+a browser without a token header, then the operator enters `server.token` in the
+page. API calls from the console still use the same token-protected management
+and sync endpoints as external clients. The console keeps the token in tab
+session storage rather than persistent browser storage.
+
+If `server.token` is configured, pass it as:
+
+```text
+Authorization: Bearer <token>
+```
+
+or:
+
+```text
+X-Api-Token: <token>
+```
+
+The server requires a token by default. An empty token is accepted only when
+`server.allow_unauthenticated_localhost: true` is explicitly configured and the
+server is bound to a loopback address.
+
+## Media Backup Semantics
+
+Backup policy separates three media modes:
+
+- `capture_text`: store message text.
+- `capture_media_metadata`: store Telegram media metadata in the message row.
+- `download_media`: download media files and expose them through `/sync/media-files`.
+
+Media files requested by `download_media: true` are stored under a `media/`
+directory next to the SQLite database. Download failures are retried according
+to `telegram.media_download` and then recorded in `/manage/operation-events`
+if they still fail.
+
+## Daily Packaging
+
+Daily packages are generated from already archived messages. The run selects
+enabled, non-removed backup origins by account, origin, topic, tag intersection,
+or tag groups, then skips origins with no messages in the selected daily
+window. Parent origins and forum topics are grouped together by the parent's
+tags unless a topic has explicit different tags or is marked important.
+When no ad hoc tag group scope is supplied, origins are grouped by their
+effective CSV tag set for package navigation and point metadata. Explicit tag
+groups are assigned from most-specific to least-specific, but unmatched origins
+still enter the all-origin point flow. Normal tag groups no longer create their
+own summary records.
+
+Origin rows can be marked `important`; important origins are packaged separately
+and analyzed in full context. Every eligible origin, including important ones,
+also participates in a separate structured message-point pipeline. Daily runs
+therefore produce two independent products:
+
+- image media analysis with OCR/visual extraction through Codex image inputs;
+- non-image long media such as PDF/video preserved as file references;
+- message-point extraction from important and non-important origins, with time,
+  tags, content, Telegram links, importance, and source references;
+- full-context analysis and a daily report sourced only from important origins;
+- a separate daily digest sourced only from the persisted message points.
+
+Package and summary artifacts are written under the configured daily output
+directory, while SQLite stores run status, paths, counts, errors, typed summary
+records, and individually queryable message points for API lookup/filtering.
+Normal point queries expose completed runs; diagnostic callers can opt into
+failed, canceled, or still-running run points explicitly.
+
+When Telegram delivery is enabled, the important report and point digest are
+sent as separate logical messages to the configured target. The point digest
+uses the fixed searchable tag `#point`; the important report keeps its source
+tags.
+
+The default Codex CLI template selects `gpt-5.6-sol` and expands task-specific
+`{model}` and `{output_schema}` placeholders before invoking the provider.
+
+API and scheduled package-plus-summary requests use the same durable SQLite job
+queue. Equivalent active or completed requests are deduplicated unless
+`force: true` or CLI `--force` is supplied. Summary records and delivery outbox
+chunks are committed atomically; delivery failures remain retryable without
+turning an already completed summary into a failed run.
+
+## Runtime Architecture
+
+- `TelegramRuntimeManager` supervises each account independently and reuses one
+  connected client for all Telegram operations.
+- `DailyJobWorker` owns package-plus-summary execution, lease recovery,
+  cancellation, and delivery-outbox draining. No background job depends on an
+  untracked daemon thread.
+- `ArchiveStore` uses WAL, busy timeouts, explicit short transactions, and one
+  SQLite connection per worker/request thread.
+- Numbered, transactional migrations upgrade the archive schema. Job and
+  outbox state transitions have database-level validation triggers.
+- The HTTP route/auth registry and request validation are driven by
+  `server/contracts.py`; generated Markdown/OpenAPI files and runtime docs share
+  the same contract hash.
+
+The default AI provider is a configurable local `codex exec` command template
+using `--output-last-message`. Templates can use `{output}`, `{images}`, and
+`{task}`. An optional `daily.ai.fallback` can switch the remainder of a run to
+an OpenAI-compatible Responses endpoint only when Codex reports a usage limit.
+The API key is read from an ignored local file; transient fallback failures can
+be durably retried once after a configured delay. Set `daily.ai.provider:
+disabled` only for local testing or dry runs.
+
+## Design Boundary
+
+The server is responsible for durable collection, sync, capture-management
+state, daily packaging, and local daily analysis jobs. Client-side features such
+as labels and app-specific UI state should live in the Mac app.
+
+`mess-end` is the host-facing manager, while `tele-mess-core` remains an
+independently runnable engine:
+
+| Owner | Responsibilities |
+| --- | --- |
+| `tele-mess-core` | Telegram sessions, SQLite schema and migrations, ingestion, capture policy, daily jobs, delivery, and optional HTTP/API service. |
+| `mess-end` | Version selection and installation, workspace selection, first-run UI, process start/stop/status, updates, and macOS LaunchAgent integration. |
+
+`mess-end` should use the public CLI and management API rather than importing
+internal Python modules or modifying the archive database directly. It should
+also enforce one core process per workspace: `run-local` and `run-server` must
+not own the same Telegram sessions and SQLite archive simultaneously. For a
+persistent managed installation, `mess-end` should point LaunchAgent at a
+stable, pinned executable rather than an incidental `uvx` cache path.
+
+## License
+
+Apache-2.0. See the
+[license](https://github.com/dreaifekks/tele-mess-core/blob/master/LICENSE).