cortexdb-connectors 0.2.20__tar.gz → 0.2.22__tar.gz

cortexdb_connectors-0.2.22/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

{cortexdb_connectors-0.2.20 → cortexdb_connectors-0.2.22}/PKG-INFO

@@ -1,13 +1,12 @@
 Metadata-Version: 2.4
 Name: cortexdb-connectors
-Version: 0.2.20
-Summary: Data connectors for CortexDB — ingest from Slack, GitHub, GitLab, Jira, Linear, Confluence, Notion, PagerDuty, Discord, Teams, Google Workspace, Salesforce, HubSpot, Zendesk, Intercom, and ServiceNow into the v1 memory API.
+Version: 0.2.22
+Summary: Data connectors for CortexDB — ingest business-system data into the v1 memory API.
 Author-email: CortexDB Team <team@cortexdb.ai>
 License-Expression: Apache-2.0
 Project-URL: Homepage, https://cortexdb.ai
 Project-URL: Documentation, https://cortexdb.ai/docs/connectors/slack
-Project-URL: Repository, https://github.com/cortexdb/cortexdb
-Project-URL: Issues, https://github.com/cortexdb/cortexdb/issues
+Project-URL: Support, https://cortexdb.ai/contact
 Keywords: ai,memory,connectors,etl,slack,github,jira,cortexdb
 Classifier: Development Status :: 4 - Beta
 Classifier: Environment :: Console
@@ -21,6 +20,7 @@ Classifier: Topic :: Database
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
+License-File: LICENSE
 Requires-Dist: httpx>=0.27
 Requires-Dist: pydantic>=2.0
 Requires-Dist: croniter>=2.0
@@ -77,6 +77,7 @@ Requires-Dist: PyYAML>=6.0; extra == "all"
 Requires-Dist: starlette>=0.37; extra == "all"
 Requires-Dist: uvicorn>=0.29; extra == "all"
 Requires-Dist: python-multipart>=0.0.9; extra == "all"
+Dynamic: license-file
 
 # cortexdb-connectors
 
@@ -123,11 +124,37 @@ Or set env vars:
 export CORTEXDB_URL=https://api-v1.cortexdb.ai
 export CORTEXDB_API_KEY=v4.public...
 export CORTEXDB_ACTOR=user:u_019e...
+export CORTEXDB_TENANT_ID=acme                            # optional assertion; verified via whoami
 export CORTEXDB_SCOPE_TEMPLATE='org:acme/source:{source}'   # optional
+export CORTEXDB_SOURCE_INSTANCE_ID='acme-slack-workspace'   # see below
 ```
 
 `{source}` (and `{scope}`, `{namespace}`, `{actor}`, `{entity.<type>}`) get filled in per event. (`{tenant}` is still accepted as a legacy alias of `{scope}`.)
 
+**Self-hosted server with a static `CORTEX_API_KEY`:** that key has no `sub`
+claim, and the server runs a request without `X-Cortex-Actor` as `user:local`.
+When `CORTEXDB_ACTOR` is unset, `cortexdb-sync sync|watch` and the Docker
+worker ask `GET /v1/auth/whoami` (without the header) and adopt `user:local`
+only when the server confirms a static key (0.2.22+). With a PASETO/JWT token
+the actor stays explicit and must equal the token's `sub`. When no actor can
+be determined the sync stops before fetching anything, with one error naming
+`CORTEXDB_ACTOR`, and the cursor is left untouched.
+
+`CORTEXDB_SOURCE_INSTANCE_ID` is an optional stable, non-secret account/site
+identifier. Set it (or `--source-instance-id`) whenever two accounts of the
+same provider write to the same CortexDB endpoint, actor, and scope. Its digest
+partitions canonical source ids and the lifecycle ledger; never put an access
+token or other credential in it. The empty default preserves existing
+single-account identities. Connector-specific YAML may set
+`source_instance_id` under that connector as well.
+
+Before it opens durable lifecycle state, the connector resolves the billing
+tenant from `GET /v1/auth/whoami`. `CORTEXDB_TENANT_ID` (or
+`--cortex-tenant-id`) is an optional expected value and must match that
+authenticated response; it never bypasses verification. Tenant identity is
+part of the opaque ledger namespace, so credentials for a different tenant
+cannot reuse completed source rows.
+
 ## Run
 
 ```bash
@@ -156,10 +183,10 @@ All 18 connectors are **🟢 Free** to self-host and **🔒 Starter+** ($29/mo)
 
 ```
 slack            SLACK_BOT_TOKEN
-github           GITHUB_TOKEN
+github           GITHUB_TOKEN, GITHUB_REPOS
 gitlab           GITLAB_TOKEN
-jira             JIRA_URL, JIRA_EMAIL, JIRA_API_TOKEN   (incl. attachments → /v1/blobs multimodal extraction, 0.2.10+)
-freshdesk        FRESHDESK_DOMAIN, FRESHDESK_API_KEY    (incl. attachments → /v1/blobs multimodal extraction, 0.2.20+)
+jira             JIRA_URL, JIRA_EMAIL, JIRA_API_TOKEN   (incl. crash-safe keyed attachments + multimodal extraction, 0.2.10+)
+freshdesk        FRESHDESK_DOMAIN, FRESHDESK_API_KEY    (incl. crash-safe keyed attachments + multimodal extraction, 0.2.20+)
 tldv             TLDV_API_KEY
 linear           LINEAR_API_KEY
 confluence       CONFLUENCE_URL, CONFLUENCE_EMAIL, CONFLUENCE_API_TOKEN
@@ -175,6 +202,44 @@ intercom         INTERCOM_TOKEN
 servicenow       SNOW_INSTANCE, SNOW_USERNAME, SNOW_PASSWORD
 ```
 
+`cortexdb-sync list` prints the same required variables plus each connector's optional ones.
+
+### GitHub
+
+```bash
+export GITHUB_TOKEN=ghp_...                  # PAT or GitHub App installation token
+export GITHUB_REPOS=acme/api,acme/web        # required: owner/repo, comma-separated (no discovery)
+export GITHUB_EVENTS=pull_request,issues     # optional; default: all six kinds below
+export GITHUB_BACKFILL_DAYS=30               # optional; first-sync history, 0 disables
+cortexdb-sync sync github
+```
+
+- **What is stored (0.2.22+):** PRs as `PR #n [state]: title` plus the full
+  description; issues with their full body; issue comments, review bodies and
+  inline review comments (file, line and diff hunk) in full; pushes with the
+  newest 20 commit messages and the total count. Nothing is truncated — text
+  over 1 MiB is split losslessly into source-bound fragments.
+- **`GITHUB_EVENTS` kinds:** `push`, `pull_request`, `issues`,
+  `issue_comment`, `pull_request_review`, `pull_request_review_comment`. Raw
+  Events API names (`PullRequestEvent`, `PullRequestReviewCommentEvent`, …)
+  are accepted too, case-insensitively.
+- **Sources:** PRs, issues, issue comments and review comments are read from
+  the REST list endpoints on every sync (the first sync reaches back
+  `GITHUB_BACKFILL_DAYS`, default 30, as does the first sync of a
+  repository added to `GITHUB_REPOS` later); the repository Events feed, which
+  GitHub caps at 300 events / 30 days, supplies pushes and reviews. Each
+  record is keyed by the GitHub object — the same identity the webhook
+  receiver uses — so a PR re-read by the poller or delivered by a webhook
+  updates one record instead of adding another.
+- **Persist `~/.cortexdb` between runs:** it holds the cursor (with the
+  work a run deferred — pushes and PR file lists / review listings held
+  when the API budget ran low, and a new repository's backfill items that
+  failed to ingest — retried on the next sync) and the
+  lifecycle ledger that retires superseded versions (or point
+  `CORTEXDB_STATE_DIR` at a volume for the ledger, and `--state-file` /
+  the worker's `CORTEX_SYNC_STATE_PATH` for the cursor). A throwaway
+  `docker run --rm` loses both.
+
 Per-connector setup pages live at https://cortexdb.ai/docs/connectors/ (e.g. [Slack](https://cortexdb.ai/docs/connectors/slack)). **The "Add Connector" button on those pages is the managed (Starter+) path** — the CLI on this page is the free path.
 
 For deep self-hosted operation — auth lifetimes, cursor state, retries, OAuth setup for Salesforce/Teams/Google Workspace — see [`SELF_HOSTING.md`](./SELF_HOSTING.md) in this directory.
@@ -186,6 +251,7 @@ If env vars get unwieldy, drop a `cortexdb-connectors.yaml` next to your sync in
 ```yaml
 slack:
   slack_bot_token: xoxb-...
+  source_instance_id: acme-slack-workspace
   channels: [C01ABCDEF, C02GHIJKL]
 
 github:
@@ -213,6 +279,8 @@ connector = SlackConnector(
 connector.bind(
     actor="user:u_019e...",
     scope_template="org:acme/source:slack/channel:{entity.channel}",
+    source_instance_id="acme-slack-workspace",
+    cortex_tenant_id="acme",  # optional assertion, verified with whoami
 )
 
 result = asyncio.run(connector.sync())
@@ -223,6 +291,45 @@ print(result.episodes_ingested, "ingested,", len(result.errors), "errors")
 
 The `cortexdb-connectors[webhooks]` extra ships a Starlette-based webhook receiver for sources that push (Slack Events API, GitHub Apps, Jira webhooks, …) instead of polling. See `cortexdb_connectors/webhooks.py`.
 
+## Durable lifecycle ledger
+
+Every canonical text or blob source is recorded in a local SQLite ledger before
+a sync cursor advances or a webhook is acknowledged. It stores only opaque
+source/version/idempotency digests, random upload tokens, and CortexDB
+event/blob ids—never source text or file bytes. The default is
+`~/.cortexdb/connector_event_ledger.sqlite3` (or under `CORTEXDB_STATE_DIR`);
+override it with `CORTEX_CONNECTOR_EVENT_LEDGER_PATH`.
+The authenticated billing tenant is included in its opaque namespace and an
+optional configured tenant assertion must match `/v1/auth/whoami`.
+
+Keep this database on durable local storage and back it up with SQLite's online
+backup API (or stop writers and copy the database together with its `-wal` and
+`-shm` files). A replica needs the same ledger to suppress replay beyond the
+server idempotency window and to erase every historical source version, so run
+one active connector writer per source namespace unless all writers share a
+correctly coordinated ledger. Same-source generation changes use bounded,
+crash-released OS advisory-lock stripes beside the SQLite file; network
+filesystems are supported only when they provide equivalent locking semantics.
+Accepted replacements retire older generations through exact derived-then-raw
+erasure before the sync cursor advances; a failed cleanup remains durable for
+retry. Rows are removed after confirmed source erasure or supersession;
+monitor disk usage as part of connector operations.
+
+New binary attachments use only CortexDB's versioned keyed-blob protocol. The
+ledger commits a random, non-secret upload token before any blob network request,
+then preflights and uploads the bytes through `/v1/blobs/idempotent`. A timeout
+or restart reuses that same token and reconciles the authoritative blob ID, so
+it cannot create a second physical object. Deletion tombstones the token through
+the matching keyed endpoint before removing the local mapping. A missing,
+disabled, malformed, or older server implementation is a hard sync failure;
+connectors never fall back to an unkeyed upload. Existing blob-ID-only ledger
+rows from releases before keyed uploads remain deletable through the legacy
+ID route as a migration path, but no new rows are created that way.
+Binary payloads over the 32 MiB source ceiling, malformed descriptors, and
+invalid envelope metadata are rejected before whoami, ledger allocation, or
+upload. Text-shaped attachments are strictly decoded and losslessly fragmented
+at UTF-8 boundaries instead of taking the binary path.
+
 ## License
 
 Apache-2.0

cortexdb_connectors-0.2.22/README.md

@@ -0,0 +1,254 @@
+# cortexdb-connectors
+
+Managed data connectors for [CortexDB](https://cortexdb.ai). Run them yourself, or let CortexDB host them for you on the [Starter tier](https://cortexdb.ai/pricing).
+
+Each connector pulls from a third-party system (Slack, GitHub, Jira, …) and writes every event into CortexDB as a v1 experience.
+
+## Plan availability (audit BLK-3)
+
+| Mode | Plan | What it means |
+|---|---|---|
+| **Self-hosted** (`cortexdb-sync` CLI) | 🟢 **Free** (this package) | You run the poller / webhook receiver yourself. Cursor state lives in `~/.cortexdb/`. Every connector below works in this mode. |
+| **Managed sync** (dashboard) | 🔒 **Starter+** ($29/mo) | CortexDB hosts the syncer, manages credentials, and surfaces health in the dashboard. The "Add Connector" button in the dashboard is gated on this plan. |
+
+The free self-hosted path is always available; the managed dashboard path is what the pricing page calls "Starter+". The badge below appears on each connector's setup page for the same reason.
+
+## Install
+
+```bash
+# core only — no third-party SDK dependencies
+pip install cortexdb-connectors
+
+# with one source's SDK
+pip install 'cortexdb-connectors[slack]'
+
+# everything
+pip install 'cortexdb-connectors[all]'
+```
+
+Requires Python 3.10+.
+
+## Auth
+
+The connectors talk to the CortexDB v1 API. Two things you need:
+
+- **A PASETO bearer token** (`Authorization: Bearer …`)
+- **An actor id** that matches the token's `sub` claim (`X-Cortex-Actor: …`)
+
+The fastest way to get both is to install [`cortexdb-cli`](https://pypi.org/project/cortexdb-cli/) and run `cortexdb init` — anonymous signup, no email or card, 7-day free-tier token. The connectors read `~/.cortexdb/state.json` automatically.
+
+Or set env vars:
+
+```bash
+export CORTEXDB_URL=https://api-v1.cortexdb.ai
+export CORTEXDB_API_KEY=v4.public...
+export CORTEXDB_ACTOR=user:u_019e...
+export CORTEXDB_TENANT_ID=acme                            # optional assertion; verified via whoami
+export CORTEXDB_SCOPE_TEMPLATE='org:acme/source:{source}'   # optional
+export CORTEXDB_SOURCE_INSTANCE_ID='acme-slack-workspace'   # see below
+```
+
+`{source}` (and `{scope}`, `{namespace}`, `{actor}`, `{entity.<type>}`) get filled in per event. (`{tenant}` is still accepted as a legacy alias of `{scope}`.)
+
+**Self-hosted server with a static `CORTEX_API_KEY`:** that key has no `sub`
+claim, and the server runs a request without `X-Cortex-Actor` as `user:local`.
+When `CORTEXDB_ACTOR` is unset, `cortexdb-sync sync|watch` and the Docker
+worker ask `GET /v1/auth/whoami` (without the header) and adopt `user:local`
+only when the server confirms a static key (0.2.22+). With a PASETO/JWT token
+the actor stays explicit and must equal the token's `sub`. When no actor can
+be determined the sync stops before fetching anything, with one error naming
+`CORTEXDB_ACTOR`, and the cursor is left untouched.
+
+`CORTEXDB_SOURCE_INSTANCE_ID` is an optional stable, non-secret account/site
+identifier. Set it (or `--source-instance-id`) whenever two accounts of the
+same provider write to the same CortexDB endpoint, actor, and scope. Its digest
+partitions canonical source ids and the lifecycle ledger; never put an access
+token or other credential in it. The empty default preserves existing
+single-account identities. Connector-specific YAML may set
+`source_instance_id` under that connector as well.
+
+Before it opens durable lifecycle state, the connector resolves the billing
+tenant from `GET /v1/auth/whoami`. `CORTEXDB_TENANT_ID` (or
+`--cortex-tenant-id`) is an optional expected value and must match that
+authenticated response; it never bypasses verification. Tenant identity is
+part of the opaque ledger namespace, so credentials for a different tenant
+cannot reuse completed source rows.
+
+## Run
+
+```bash
+# one shot
+cortexdb-sync sync slack
+
+# poll loop, 60s between cycles
+cortexdb-sync watch slack --interval 60
+
+# resume from a specific point
+cortexdb-sync sync github --since 2026-05-01T00:00:00Z
+
+# see what's available
+cortexdb-sync list
+
+# check resolved CortexDB creds
+cortexdb-sync auth
+
+# cursor state across every connector
+cortexdb-sync status
+```
+
+## Available connectors
+
+All 18 connectors are **🟢 Free** to self-host and **🔒 Starter+** ($29/mo) to use via the managed dashboard.
+
+```
+slack            SLACK_BOT_TOKEN
+github           GITHUB_TOKEN, GITHUB_REPOS
+gitlab           GITLAB_TOKEN
+jira             JIRA_URL, JIRA_EMAIL, JIRA_API_TOKEN   (incl. crash-safe keyed attachments + multimodal extraction, 0.2.10+)
+freshdesk        FRESHDESK_DOMAIN, FRESHDESK_API_KEY    (incl. crash-safe keyed attachments + multimodal extraction, 0.2.20+)
+tldv             TLDV_API_KEY
+linear           LINEAR_API_KEY
+confluence       CONFLUENCE_URL, CONFLUENCE_EMAIL, CONFLUENCE_API_TOKEN
+notion           NOTION_TOKEN
+pagerduty        PAGERDUTY_API_KEY
+discord          DISCORD_BOT_TOKEN
+teams            TEAMS_TENANT_ID, TEAMS_CLIENT_ID, TEAMS_CLIENT_SECRET
+google-workspace GW_SERVICE_ACCOUNT_KEY, GW_DELEGATED_USER
+salesforce       SF_INSTANCE_URL, SF_CLIENT_ID, SF_CLIENT_SECRET, SF_USERNAME, SF_PASSWORD
+hubspot          HUBSPOT_TOKEN
+zendesk          ZENDESK_SUBDOMAIN, ZENDESK_EMAIL, ZENDESK_TOKEN
+intercom         INTERCOM_TOKEN
+servicenow       SNOW_INSTANCE, SNOW_USERNAME, SNOW_PASSWORD
+```
+
+`cortexdb-sync list` prints the same required variables plus each connector's optional ones.
+
+### GitHub
+
+```bash
+export GITHUB_TOKEN=ghp_...                  # PAT or GitHub App installation token
+export GITHUB_REPOS=acme/api,acme/web        # required: owner/repo, comma-separated (no discovery)
+export GITHUB_EVENTS=pull_request,issues     # optional; default: all six kinds below
+export GITHUB_BACKFILL_DAYS=30               # optional; first-sync history, 0 disables
+cortexdb-sync sync github
+```
+
+- **What is stored (0.2.22+):** PRs as `PR #n [state]: title` plus the full
+  description; issues with their full body; issue comments, review bodies and
+  inline review comments (file, line and diff hunk) in full; pushes with the
+  newest 20 commit messages and the total count. Nothing is truncated — text
+  over 1 MiB is split losslessly into source-bound fragments.
+- **`GITHUB_EVENTS` kinds:** `push`, `pull_request`, `issues`,
+  `issue_comment`, `pull_request_review`, `pull_request_review_comment`. Raw
+  Events API names (`PullRequestEvent`, `PullRequestReviewCommentEvent`, …)
+  are accepted too, case-insensitively.
+- **Sources:** PRs, issues, issue comments and review comments are read from
+  the REST list endpoints on every sync (the first sync reaches back
+  `GITHUB_BACKFILL_DAYS`, default 30, as does the first sync of a
+  repository added to `GITHUB_REPOS` later); the repository Events feed, which
+  GitHub caps at 300 events / 30 days, supplies pushes and reviews. Each
+  record is keyed by the GitHub object — the same identity the webhook
+  receiver uses — so a PR re-read by the poller or delivered by a webhook
+  updates one record instead of adding another.
+- **Persist `~/.cortexdb` between runs:** it holds the cursor (with the
+  work a run deferred — pushes and PR file lists / review listings held
+  when the API budget ran low, and a new repository's backfill items that
+  failed to ingest — retried on the next sync) and the
+  lifecycle ledger that retires superseded versions (or point
+  `CORTEXDB_STATE_DIR` at a volume for the ledger, and `--state-file` /
+  the worker's `CORTEX_SYNC_STATE_PATH` for the cursor). A throwaway
+  `docker run --rm` loses both.
+
+Per-connector setup pages live at https://cortexdb.ai/docs/connectors/ (e.g. [Slack](https://cortexdb.ai/docs/connectors/slack)). **The "Add Connector" button on those pages is the managed (Starter+) path** — the CLI on this page is the free path.
+
+For deep self-hosted operation — auth lifetimes, cursor state, retries, OAuth setup for Salesforce/Teams/Google Workspace — see [`SELF_HOSTING.md`](./SELF_HOSTING.md) in this directory.
+
+## YAML config
+
+If env vars get unwieldy, drop a `cortexdb-connectors.yaml` next to your sync invocation:
+
+```yaml
+slack:
+  slack_bot_token: xoxb-...
+  source_instance_id: acme-slack-workspace
+  channels: [C01ABCDEF, C02GHIJKL]
+
+github:
+  github_token: ghp_...
+  repos: [acme/api, acme/web]
+  events: [pull_request, issue_comment]
+```
+
+## Programmatic use
+
+Skip the CLI entirely:
+
+```python
+import asyncio
+from cortexdb_connectors.slack import SlackConnector
+
+connector = SlackConnector(
+    cortex_url="https://api-v1.cortexdb.ai",
+    cortex_api_key="v4.public...",
+    scope="org:acme/source:slack",
+    slack_bot_token="xoxb-...",
+    channels=["C01ABC", "C02DEF"],
+)
+# actor + a richer scope_template are set after construction:
+connector.bind(
+    actor="user:u_019e...",
+    scope_template="org:acme/source:slack/channel:{entity.channel}",
+    source_instance_id="acme-slack-workspace",
+    cortex_tenant_id="acme",  # optional assertion, verified with whoami
+)
+
+result = asyncio.run(connector.sync())
+print(result.episodes_ingested, "ingested,", len(result.errors), "errors")
+```
+
+## Webhooks
+
+The `cortexdb-connectors[webhooks]` extra ships a Starlette-based webhook receiver for sources that push (Slack Events API, GitHub Apps, Jira webhooks, …) instead of polling. See `cortexdb_connectors/webhooks.py`.
+
+## Durable lifecycle ledger
+
+Every canonical text or blob source is recorded in a local SQLite ledger before
+a sync cursor advances or a webhook is acknowledged. It stores only opaque
+source/version/idempotency digests, random upload tokens, and CortexDB
+event/blob ids—never source text or file bytes. The default is
+`~/.cortexdb/connector_event_ledger.sqlite3` (or under `CORTEXDB_STATE_DIR`);
+override it with `CORTEX_CONNECTOR_EVENT_LEDGER_PATH`.
+The authenticated billing tenant is included in its opaque namespace and an
+optional configured tenant assertion must match `/v1/auth/whoami`.
+
+Keep this database on durable local storage and back it up with SQLite's online
+backup API (or stop writers and copy the database together with its `-wal` and
+`-shm` files). A replica needs the same ledger to suppress replay beyond the
+server idempotency window and to erase every historical source version, so run
+one active connector writer per source namespace unless all writers share a
+correctly coordinated ledger. Same-source generation changes use bounded,
+crash-released OS advisory-lock stripes beside the SQLite file; network
+filesystems are supported only when they provide equivalent locking semantics.
+Accepted replacements retire older generations through exact derived-then-raw
+erasure before the sync cursor advances; a failed cleanup remains durable for
+retry. Rows are removed after confirmed source erasure or supersession;
+monitor disk usage as part of connector operations.
+
+New binary attachments use only CortexDB's versioned keyed-blob protocol. The
+ledger commits a random, non-secret upload token before any blob network request,
+then preflights and uploads the bytes through `/v1/blobs/idempotent`. A timeout
+or restart reuses that same token and reconciles the authoritative blob ID, so
+it cannot create a second physical object. Deletion tombstones the token through
+the matching keyed endpoint before removing the local mapping. A missing,
+disabled, malformed, or older server implementation is a hard sync failure;
+connectors never fall back to an unkeyed upload. Existing blob-ID-only ledger
+rows from releases before keyed uploads remain deletable through the legacy
+ID route as a migration path, but no new rows are created that way.
+Binary payloads over the 32 MiB source ceiling, malformed descriptors, and
+invalid envelope metadata are rejected before whoami, ledger allocation, or
+upload. Text-shaped attachments are strictly decoded and losslessly fragmented
+at UTF-8 boundaries instead of taking the binary path.
+
+## License
+
+Apache-2.0