context-use 0.1.0__tar.gz

context_use-0.1.0/.gitignore

@@ -0,0 +1,23 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+
+data/**
+!data/*/
+!data/*/.gitkeep
+
+*.db
+*.zip
+
+# Environment files
+.env*
+!.env*.example
+
+.DS_Store

context_use-0.1.0/CHANGELOG.md

@@ -0,0 +1,125 @@
+## [0.1.0] - 2026-03-09
+
+### 🚀 Features
+
+- Add Google ETL for Discover and Lens (#116)
+- Centralize memory operations (#115)
+- Add Google Shopping ETL (#114)
+- Add ETL for Google Youtube (#113)
+- Add Google search ETLs (#112)
+- Claude conversations etl and memory pipelines (#111)
+- Add FibreDislike (#109)
+- Provider and interaction auto registration (#107)
+- Remove ask command (#106)
+- Pg as an optional dep (#104)
+- Split cli commands (#101)
+- Basic profile generation skill (#100)
+- Remove one shot user profile generation (#99)
+- Agent with memory synthesise skill (#98)
+- Refinement agent (#96)
+- Remove memory refinement pipeline (#95)
+- Remove unused store methods (#94)
+- [**breaking**] Generate memories from unprocessed threads (#93)
+- Bump thread payload version to 1.1.0 (#91)
+- Add IG ETL saved posts and collections (#90)
+- Add IG ETL for likes, followers and comments (#89)
+- Add IG ETL for posts_viewed and profile_searches (#88)
+- Add fibre classes needed for instagram and google (#87)
+- Add complete ActivityStreams schemas (#86)
+- Add IG ETL for videos watched (#83)
+- [**breaking**] Switch EtlTask to plural source_uris (#79)
+- BatchContext to group configs (#78)
+- Resolve asset uri for gcs storage (#77)
+- Cli command to run e2e batch pipeline (#71)
+- Config registry (#76)
+- Base llm client and litellm implementation (#74)
+- Cli quickstart and pg config (#68)
+- Common categories for prompts (#67)
+- Real time mode on subset of the data (#60)
+- Use store facade (#57)
+- Pg store (#56)
+- In memory store (#55)
+- Add abstract storage models (#54)
+- Group ids instead of key (#53)
+- Decouple batches and etls (#52)
+- Add AGENTS.md doc for building ETL, symlink from CLAUDE and CONTRIBUTING mds (#42)
+- Process chatgpt conversations (#44)
+- Thread grouping (#43)
+- Switch IG media ETL to pipe for ET and loader for L (#37)
+- Switch gpt ETL to pipe for ET and loader for L (#36)
+- Add foundation classes for Pipe ET pipeline and a separate Loader (#35)
+- Mcp-use extension (#33)
+- Support for latest openai models (#31)
+- Enable pgvector extension (#27)
+- Batch memory embeddings (#25)
+- Litellm as llm client (#24)
+- Gemini batch api implementation (#22)
+- Dedup threads in upload strategy (#21)
+- Simple batch memories generation from ig media (#18)
+- Add etl_tasks.source_uri and archives.file_uris columns (#15)
+- Only use postgres as db (#13)
+- Basic etl from zip archives (#5)
+- Initial repo setup
+
+### 🐛 Bug Fixes
+
+- Report extraction failures from pipe (#117)
+- Return full agent response (#103)
+- IG ETL liked_posts v1/v0 split (#97)
+- Allow IG profile_searches without username as value (#92)
+- Make pipe resilient to parsing errors (#85)
+- Fix videos_watches parsing (#84)
+- Switch pipe versions to numeric (#81)
+- Rm unused EtlTaskStatus fields (#80)
+- Process split chatgpt converations (#72)
+- Switch to legacy format for thread payload (#70)
+- Simplify AGENTS.md, remove code quotations (#66)
+- Use unique key suffix from payload as unique_key (#65)
+- Archives concurrency (#51)
+- Switch pipe archive path to glob pattern matching (#41)
+- Absolute path resolution in storage backend (#28)
+- Make asset_uri have the path relative to DiskStorage root (#14)
+
+### 💼 Other
+
+- Downgrade asyncpg to 0.30.0 (#69)
+- [**breaking**] Lower minimum python to 3.12 (#62)
+
+### 🚜 Refactor
+
+- Agent module outside of memories (#108)
+- Move configs out of cli (#102)
+- Consolidate places where we generate a uuid (#63)
+- Unify etl and memory configs (#45)
+- Add pydantic-typed records between E and T (#23)
+- [**breaking**] Async db (#29)
+- [**breaking**] Async litellm (#26)
+- Rm task descriptors, use EtlTask (#19)
+- Rm TaskMetadata, replace with EtlTask (#17)
+- Move session ownership out of etl (#16)
+- Etl module (#10)
+
+### 🧪 Testing
+
+- Refactor tests folder (#61)
+- Add pipe test kit which provides generic test harness for new etl (#40)
+- Add synthetic archives for ig and gpt data (#30)
+
+### ⚙️ Miscellaneous Tasks
+
+- Only include necessary files in build (#126)
+- Remove code-specific guides from contributing (#125)
+- Publish release workflow (#124)
+- Prepare release workflow (#119)
+- Generate changelog using `git-cliff` (#118)
+- Exit at first failure (#110)
+- Update readme (#105)
+- Rm stale ETL Loader classes (#73)
+- [**breaking**] Drop pandas from dependencies (#39)
+- Clean up dead ETL code (#38)
+- Remove useless comments (#34)
+- Typecheck tests (#20)
+- Minor (#12)
+- Docker compose local env (#11)
+- Run pytest on every PR (#8)
+- Pre-commit config (#9)

context_use-0.1.0/PKG-INFO

@@ -0,0 +1,176 @@
+Metadata-Version: 2.4
+Name: context-use
+Version: 0.1.0
+Requires-Python: >=3.12
+Requires-Dist: ijson>=3.0
+Requires-Dist: litellm>=1.81.13
+Requires-Dist: pydantic>=2.0
+Requires-Dist: tenacity>=9.1.4
+Provides-Extra: adk
+Requires-Dist: google-adk>=1.26; extra == 'adk'
+Provides-Extra: gcs
+Requires-Dist: google-cloud-storage<3,>=2.10.0; extra == 'gcs'
+Provides-Extra: mcp-use
+Requires-Dist: langchain-openai>=1.1.10; extra == 'mcp-use'
+Requires-Dist: mcp-use>=1.6.0; extra == 'mcp-use'
+Provides-Extra: postgres
+Requires-Dist: asyncpg>=0.30.0; extra == 'postgres'
+Requires-Dist: greenlet>=3.3.1; extra == 'postgres'
+Requires-Dist: pgvector>=0.4.2; extra == 'postgres'
+Requires-Dist: sqlalchemy>=2.0; extra == 'postgres'
+Description-Content-Type: text/markdown
+
+# context-use
+
+Turn your data exports into portable AI memory.
+
+## Features
+
+- **Ingest** — parse provider export ZIPs into structured threads; no cloud upload required
+- **Quickstart** — zero-database preview mode; results written to `data/output/` with no setup beyond an OpenAI key
+- **Full pipeline** — persistent storage in PostgreSQL with pgvector; full archive history, batch API for cost-efficient memory generation
+- **Semantic search** — `memories search` queries your memory store by meaning, not just keywords
+- **MCP server** — expose memories and semantic search to Claude Desktop, Cursor, or any MCP client
+- **Personal agent** — multi-turn agent that synthesises higher-level pattern memories, generates a first-person profile, or runs ad-hoc queries against your memory store
+
+## Supported providers
+
+| Provider | Status | Data types | Export guide |
+|----------|--------|------------|-------------|
+| ChatGPT | Available | Conversations | [Export your data](https://help.openai.com/en/articles/7260999-how-do-i-export-my-chatgpt-history-and-data) |
+| Instagram | Available | Stories, Reels, Posts | [Download your data](https://help.instagram.com/181231772500920) |
+| WhatsApp | Coming soon | | |
+| Google Takeout | Coming soon | | |
+
+## Getting your export
+
+1. Follow the export guide for your provider in the table above. The export is delivered as a ZIP file — **do not extract it**.
+2. Move or copy the ZIP into `data/input/` inside the cloned repo:
+
+```
+context-use/
+└── data/
+    └── input/
+        └── chatgpt-export.zip   ← place it here
+```
+
+Both `quickstart` and `pipeline` scan `data/input/` for exports on startup and prompt you to pick one if multiple are present.
+
+## Install
+
+```bash
+git clone https://github.com/onfabric/context-use.git
+cd context-use
+uv sync
+source .venv/bin/activate
+```
+
+Set your OpenAI API key:
+
+```bash
+context-use config set-key
+# or: export OPENAI_API_KEY=sk-...
+```
+
+## Quick start
+
+A zero-setup preview that requires no database.
+
+```bash
+context-use quickstart
+```
+
+The CLI prompts for the export and provider. Memory generation uses the OpenAI **real-time API** — fast for small slices but susceptible to rate limits on large exports. By default only the last 30 days are processed; use `--full` to include the complete history (the CLI warns you before proceeding).
+
+The output is a snapshot: memories are written to `data/output/` as Markdown and JSON, then discarded. Nothing is stored in a database, so the memories are not queryable, searchable, or available to the MCP server after the command exits.
+
+**The full pipeline is the intended way to use context-use beyond this initial preview.**
+
+## Full pipeline
+
+For persistent storage, semantic search, and the MCP server.
+
+**1. Set up PostgreSQL (one-time)**
+
+```bash
+context-use config set-store postgres
+```
+
+Prompts to start a local container via Docker, then saves connection details to `~/.config/context-use/config.toml`. Skip Docker if you're bringing your own PostgreSQL instance.
+
+**2. Run the pipeline**
+
+```bash
+context-use pipeline
+```
+
+Ingests the export and generates memories via the OpenAI **batch API** — significantly cheaper and more rate-limit-friendly than the real-time API used by quickstart. Typical runtime: 2–10 minutes. Memories are stored in PostgreSQL and persist across sessions, enabling semantic search, the MCP server, and the personal agent.
+
+**3. Explore your memories**
+
+```bash
+context-use memories list
+context-use memories search "hiking trips in 2024"
+```
+
+## MCP server
+
+Requires the full pipeline (PostgreSQL).
+
+```bash
+python -m context_use.ext.mcp_use.run
+# use --transport stdio for clients that prefer stdio
+```
+
+Add to your MCP client config (Claude Desktop, Cursor, etc.):
+
+```json
+{
+  "mcpServers": {
+    "context-use": {
+      "command": "python",
+      "args": ["-m", "context_use.ext.mcp_use.run", "--transport", "stdio"]
+    }
+  }
+}
+```
+
+Claude Desktop config path: `~/Library/Application Support/Claude/claude_desktop_config.json`. Cursor: Settings → MCP.
+
+## Personal agent
+
+A multi-turn agent that operates over your full memory store. Requires PostgreSQL.
+
+```bash
+context-use config set-agent adk
+context-use agent synthesise          # generate higher-level pattern memories
+context-use agent profile             # compile a first-person profile
+context-use agent ask "What topics do I keep coming back to across all my conversations?"
+```
+
+## Configuration
+
+Config file: `~/.config/context-use/config.toml`. Run `context-use config show` to see all active values and where each comes from (env var, file, or built-in default).
+
+| Setting | CLI command | Env var | Default |
+|---------|-------------|---------|---------|
+| OpenAI API key | `config set-key` | `OPENAI_API_KEY` | — |
+| Model | edit config file | `OPENAI_MODEL` | `gpt-5.2` |
+| Embedding model | edit config file | `OPENAI_EMBEDDING_MODEL` | `text-embedding-3-large` |
+| Store backend | `config set-store postgres\|memory` | `CONTEXT_USE_STORE` | `memory` |
+| PostgreSQL host | `config set-store postgres` | `POSTGRES_HOST` | `localhost` |
+| PostgreSQL port | `config set-store postgres` | `POSTGRES_PORT` | `5432` |
+| PostgreSQL database | `config set-store postgres` | `POSTGRES_DB` | `context_use` |
+| PostgreSQL user | `config set-store postgres` | `POSTGRES_USER` | `postgres` |
+| PostgreSQL password | `config set-store postgres` | `POSTGRES_PASSWORD` | `postgres` |
+| Agent backend | `config set-agent adk` | `CONTEXT_USE_AGENT_BACKEND` | — |
+| Data directory | edit config file | — | `./data` |
+
+
+## Adding new providers and pipes
+
+See [AGENTS.md](AGENTS.md) for `context-use`'s architecture and how to add new providers and pipes.
+
+## Contributing
+
+See [CONTRIBUTING.md](.github/CONTRIBUTING.md) for how to contribute to the `context-use` project.

context_use-0.1.0/README.md

@@ -0,0 +1,154 @@
+# context-use
+
+Turn your data exports into portable AI memory.
+
+## Features
+
+- **Ingest** — parse provider export ZIPs into structured threads; no cloud upload required
+- **Quickstart** — zero-database preview mode; results written to `data/output/` with no setup beyond an OpenAI key
+- **Full pipeline** — persistent storage in PostgreSQL with pgvector; full archive history, batch API for cost-efficient memory generation
+- **Semantic search** — `memories search` queries your memory store by meaning, not just keywords
+- **MCP server** — expose memories and semantic search to Claude Desktop, Cursor, or any MCP client
+- **Personal agent** — multi-turn agent that synthesises higher-level pattern memories, generates a first-person profile, or runs ad-hoc queries against your memory store
+
+## Supported providers
+
+| Provider | Status | Data types | Export guide |
+|----------|--------|------------|-------------|
+| ChatGPT | Available | Conversations | [Export your data](https://help.openai.com/en/articles/7260999-how-do-i-export-my-chatgpt-history-and-data) |
+| Instagram | Available | Stories, Reels, Posts | [Download your data](https://help.instagram.com/181231772500920) |
+| WhatsApp | Coming soon | | |
+| Google Takeout | Coming soon | | |
+
+## Getting your export
+
+1. Follow the export guide for your provider in the table above. The export is delivered as a ZIP file — **do not extract it**.
+2. Move or copy the ZIP into `data/input/` inside the cloned repo:
+
+```
+context-use/
+└── data/
+    └── input/
+        └── chatgpt-export.zip   ← place it here
+```
+
+Both `quickstart` and `pipeline` scan `data/input/` for exports on startup and prompt you to pick one if multiple are present.
+
+## Install
+
+```bash
+git clone https://github.com/onfabric/context-use.git
+cd context-use
+uv sync
+source .venv/bin/activate
+```
+
+Set your OpenAI API key:
+
+```bash
+context-use config set-key
+# or: export OPENAI_API_KEY=sk-...
+```
+
+## Quick start
+
+A zero-setup preview that requires no database.
+
+```bash
+context-use quickstart
+```
+
+The CLI prompts for the export and provider. Memory generation uses the OpenAI **real-time API** — fast for small slices but susceptible to rate limits on large exports. By default only the last 30 days are processed; use `--full` to include the complete history (the CLI warns you before proceeding).
+
+The output is a snapshot: memories are written to `data/output/` as Markdown and JSON, then discarded. Nothing is stored in a database, so the memories are not queryable, searchable, or available to the MCP server after the command exits.
+
+**The full pipeline is the intended way to use context-use beyond this initial preview.**
+
+## Full pipeline
+
+For persistent storage, semantic search, and the MCP server.
+
+**1. Set up PostgreSQL (one-time)**
+
+```bash
+context-use config set-store postgres
+```
+
+Prompts to start a local container via Docker, then saves connection details to `~/.config/context-use/config.toml`. Skip Docker if you're bringing your own PostgreSQL instance.
+
+**2. Run the pipeline**
+
+```bash
+context-use pipeline
+```
+
+Ingests the export and generates memories via the OpenAI **batch API** — significantly cheaper and more rate-limit-friendly than the real-time API used by quickstart. Typical runtime: 2–10 minutes. Memories are stored in PostgreSQL and persist across sessions, enabling semantic search, the MCP server, and the personal agent.
+
+**3. Explore your memories**
+
+```bash
+context-use memories list
+context-use memories search "hiking trips in 2024"
+```
+
+## MCP server
+
+Requires the full pipeline (PostgreSQL).
+
+```bash
+python -m context_use.ext.mcp_use.run
+# use --transport stdio for clients that prefer stdio
+```
+
+Add to your MCP client config (Claude Desktop, Cursor, etc.):
+
+```json
+{
+  "mcpServers": {
+    "context-use": {
+      "command": "python",
+      "args": ["-m", "context_use.ext.mcp_use.run", "--transport", "stdio"]
+    }
+  }
+}
+```
+
+Claude Desktop config path: `~/Library/Application Support/Claude/claude_desktop_config.json`. Cursor: Settings → MCP.
+
+## Personal agent
+
+A multi-turn agent that operates over your full memory store. Requires PostgreSQL.
+
+```bash
+context-use config set-agent adk
+context-use agent synthesise          # generate higher-level pattern memories
+context-use agent profile             # compile a first-person profile
+context-use agent ask "What topics do I keep coming back to across all my conversations?"
+```
+
+## Configuration
+
+Config file: `~/.config/context-use/config.toml`. Run `context-use config show` to see all active values and where each comes from (env var, file, or built-in default).
+
+| Setting | CLI command | Env var | Default |
+|---------|-------------|---------|---------|
+| OpenAI API key | `config set-key` | `OPENAI_API_KEY` | — |
+| Model | edit config file | `OPENAI_MODEL` | `gpt-5.2` |
+| Embedding model | edit config file | `OPENAI_EMBEDDING_MODEL` | `text-embedding-3-large` |
+| Store backend | `config set-store postgres\|memory` | `CONTEXT_USE_STORE` | `memory` |
+| PostgreSQL host | `config set-store postgres` | `POSTGRES_HOST` | `localhost` |
+| PostgreSQL port | `config set-store postgres` | `POSTGRES_PORT` | `5432` |
+| PostgreSQL database | `config set-store postgres` | `POSTGRES_DB` | `context_use` |
+| PostgreSQL user | `config set-store postgres` | `POSTGRES_USER` | `postgres` |
+| PostgreSQL password | `config set-store postgres` | `POSTGRES_PASSWORD` | `postgres` |
+| Agent backend | `config set-agent adk` | `CONTEXT_USE_AGENT_BACKEND` | — |
+| Data directory | edit config file | — | `./data` |
+
+
+## Adding new providers and pipes
+
+See [AGENTS.md](AGENTS.md) for `context-use`'s architecture and how to add new providers and pipes.
+
+## Contributing
+
+See [CONTRIBUTING.md](.github/CONTRIBUTING.md) for how to contribute to the `context-use` project.

context_use-0.1.0/context_use/__init__.py

@@ -0,0 +1,24 @@
+"""Public API for the context_use library.
+
+External consumers (CLI, MCP server, etc.) should import exclusively
+from this module.  Only unit / integration tests may reach into
+sub-packages directly.
+"""
+
+import context_use.providers  # noqa: F401 — triggers provider registration
+from context_use.facade import (
+    ContextUse,
+    PipelineResult,
+    ScheduleInstruction,
+    TaskBreakdown,
+)
+from context_use.store import InMemoryStore, Store
+
+__all__ = [
+    "ContextUse",
+    "InMemoryStore",
+    "PipelineResult",
+    "ScheduleInstruction",
+    "Store",
+    "TaskBreakdown",
+]

context_use-0.1.0/context_use/activitystreams/__init__.py

@@ -0,0 +1,170 @@
+"""
+ActivityStreams 2.0 Pydantic Models
+
+This package provides comprehensive Pydantic models for all ActivityStreams 2.0 types
+as defined in the W3C ActivityStreams 2.0 Vocabulary specification.
+
+The models are strictly compliant with the W3C specification and include:
+- Core Types: Object, Link, Activity, Collection, etc.
+- Activity Types: Accept, Add, Announce, Create, etc.
+- Actor Types: Person, Organization, Service, etc.
+- Object Types: Article, Note, Image, Video, etc.
+- Link Types: Link, Mention, and relationship types
+
+Usage:
+    from activitystreams import Note, Create, Person
+
+    # Create a note
+    note = Note(
+        type="Note",
+        content="Hello, ActivityStreams!",
+        attributedTo="https://example.com/users/alice"
+    )
+
+    # Create an activity
+    activity = Create(
+        type="Create",
+        actor="https://example.com/users/alice",
+        object=note
+    )
+"""
+
+# Core types
+# Activity types
+from .activities import (
+    Accept,
+    Add,
+    Announce,
+    Arrive,
+    Block,
+    Create,
+    Delete,
+    Dislike,
+    Flag,
+    Follow,
+    Ignore,
+    Invite,
+    Join,
+    Leave,
+    Like,
+    Listen,
+    Move,
+    Offer,
+    Question,
+    Read,
+    Reject,
+    Remove,
+    TentativeAccept,
+    TentativeReject,
+    Travel,
+    Undo,
+    Update,
+    View,
+)
+
+# Actor types
+from .actors import (
+    Actor,
+    Application,
+    Group,
+    Organization,
+    Person,
+    Service,
+)
+from .core import (
+    Activity,
+    ASType,
+    Collection,
+    CollectionPage,
+    IntransitiveActivity,
+    Link,
+    Object,
+    OrderedCollection,
+    OrderedCollectionPage,
+)
+
+# Link types and relationship types
+from .links import (
+    Mention,
+)
+
+# Object types
+from .objects import (
+    Article,
+    Audio,
+    Document,
+    Event,
+    Image,
+    Note,
+    Page,
+    Place,
+    Profile,
+    Relationship,
+    Tombstone,
+    Video,
+)
+
+# All exports organized by category
+__all__ = [
+    # Core types
+    "ASType",
+    "Object",
+    "Link",
+    "Activity",
+    "IntransitiveActivity",
+    "Collection",
+    "OrderedCollection",
+    "CollectionPage",
+    "OrderedCollectionPage",
+    # Activity types
+    "Accept",
+    "TentativeAccept",
+    "Add",
+    "Announce",
+    "Arrive",
+    "Block",
+    "Create",
+    "Delete",
+    "Dislike",
+    "Flag",
+    "Follow",
+    "Ignore",
+    "Invite",
+    "Join",
+    "Leave",
+    "Like",
+    "Listen",
+    "Move",
+    "Offer",
+    "Question",
+    "Reject",
+    "TentativeReject",
+    "Read",
+    "Remove",
+    "Travel",
+    "Undo",
+    "Update",
+    "View",
+    # Actor types
+    "Actor",
+    "Application",
+    "Group",
+    "Organization",
+    "Person",
+    "Service",
+    # Object types
+    "Article",
+    "Audio",
+    "Document",
+    "Event",
+    "Image",
+    "Note",
+    "Page",
+    "Place",
+    "Profile",
+    "Relationship",
+    "Tombstone",
+    "Video",
+    # Link types and relationship types
+    "Mention",
+]