openoutsend 0.1.16__py3-none-any.whl

cold_outreach/README.md

@@ -0,0 +1,71 @@
+# cold_outreach — ported from OpenOutreach, not yet wired up
+
+This directory is **parked code**, and it is now the whole of this repo. Nothing imports it,
+nothing runs it, and it is not on `INSTALLED_APPS`. It is the cold-outreach half of
+[OpenOutreach](https://github.com/eracle/OpenOutreach), moved here whole so it can be integrated
+deliberately rather than rewritten from memory.
+
+## Why it moved
+
+OpenOutreach became a **lead finder**: product description in, qualified leads out, with the reason
+each one was chosen, handed over as CSV. Sending is a specialism it was not good at, and a good lead
+list ruined by a bad opener reads to the buyer as a bad lead list. So the sending half comes here,
+where the whole job is the message.
+
+The boundary between the two is **one-way and public**: OpenOutreach exports leads, and nothing flows
+back. Whatever this code becomes, it must read that same export — no shared model, no direct import,
+no DB join. OpenOutSend gets no privileged path that Instantly or Smartlead would not get.
+
+The design decisions behind that split live in `roadmap/p1-e3-leadfinder-sequencer-boundary.md` in the
+`openoutreach-docs` repo. Read it before integrating; several decisions were reversed after the fact
+and the reasoning is recorded there rather than in either repo's git history.
+
+## What is here
+
+Layout mirrors OpenOutreach's, so the original import paths still read correctly.
+
+| path | what it is |
+|---|---|
+| `emails/models/mailbox.py` | `Mailbox` — one SMTP sending inbox, plus the pool-level pacing manager (`free_for_first_email`, `remaining_today`) and the measured daily cap. |
+| `emails/models/maillog.py` | `Message` / `Thread` / `DeliveryEvent` / `FolderCoverage` — the mail log. The record and its interpretation are separate objects; `processed_at IS NULL` is the third state. |
+| `emails/sync.py`, `classify.py`, `project.py`, `mail_pass.py` | The mail pass, in its three ordered jobs: sync (IMAP → rows, the only network step), classify (pure, versioned, over stored bytes), project (bounces → `DeliveryEvent`, opt-outs → suppression). |
+| `emails/threads.py` | Union-find over Message-IDs, so a reply carrying only `In-Reply-To` still reaches its deal. |
+| `emails/parsing.py` | Message parsing helpers. |
+| `emails/sender.py`, `smtp.py` | The send path: body → signature → opt-out line → attribution, `List-Unsubscribe`, SMTP auth. |
+| `emails/warmth.py`, `delivery_policy.py`, `report.py` | The three send guards — hours, rate, volume — and the receiver feedback that moves them. The daily cap is **measured** from the box's own Sent folder, not configured. |
+| `emails/steps/send.py`, `steps/reply.py` | The two pipeline steps: send one opener, answer one reply. |
+| `emails/admin.py`, `apps.py`, `migrations/` | Django wiring and the full migration history of the `emails` app. |
+| `core/agents/outreach.py`, `prompt.py` | The single outreach agent that owns the whole thread — cold open and every reply. Its job is **Mom Test research, not selling**: learn how the lead works today, never pitch unprompted. |
+| `core/templates/prompts/outreach_agent.j2` | That agent's prompt. |
+| `core/sending_window.py` | Mon–Fri, 08:00–20:00 in the *operator's* timezone, derived from their onboarding country. Openers wait for it; replies never come through it. |
+| `tests/` | The suite for all of the above, moved unchanged. |
+
+## What it still expects, and does not find here
+
+Every one of these is an OpenOutreach import that did **not** come along. They are the integration
+work, and most of them are the boundary in disguise:
+
+| import | what it was | what it has to become |
+|---|---|---|
+| `openoutreach.crm.models` (`Deal`, `Lead`, `DealState`) | the lead, the campaign-scoped deal, the state machine | **the port's main question.** A sequencer's unit is a recipient in a sequence, not a `Deal` in a finder's FSM. Do not recreate the finder's states here — take the exported row (`email, first_name, last_name, company, title, website, linkedin_url, reason, lead_id`) as the input and model the rest yourself. `lead_id` is the finder's join key; carry it, don't resolve it. |
+| `openoutreach.core.db.leads` (`suppress_email`) | permanent, account-level opt-out on `Lead.disqualified` | **the opt-out duty moves with this code.** A finder that never contacts anyone is not the sender under CAN-SPAM / GDPR / CASL, so the `List-Unsubscribe` header, the `+unsub` alias, the mailbox scan and the agent's `suppress` action are all this side's obligation now. Suppression needs a home here. |
+| `openoutreach.core.db.deals`, `db.summaries` | deal creation and conversation summaries | conversation state, so it belongs on this side; re-model against whatever replaces `Deal`. |
+| `openoutreach.core.llm` | the pydantic-ai model factory (`SiteConfig.ai_model` + key) | needs an equivalent, or accept an injected agent. |
+| `openoutreach.core.models.SiteConfig` | the config singleton — `country_code` drives the sending window | `sending_window.py` reads it directly; give it a seam rather than a singleton. |
+| `openoutreach.core.conf` | `WARM_*`, `MIN_SEND_INTERVAL_SECONDS`, `SEND_WINDOW_*` | copy the constants across; `WARM_CEILING_SENDS` is **derived** from the window and the pacing, so keep the arithmetic rather than re-declaring a number. |
+| `openoutreach.core.business_time` | Mon–Fri, used by the sending window and by thread age | small and self-contained; copy it. |
+| `openoutreach.core.operator` | the operator `User` — the BCC target on their own campaigns | an account concept this app already has in some form. |
+| `openoutreach.core.cycle` | the daemon loop that called these steps | **do not port.** The loop is the finder's, and it deleted its own daemon in favour of one bounded verb behind a timer. Sending here wants a cadence, not a resident process. |
+| `openoutreach.core.logblock` | log formatting | cosmetic; drop or replace. |
+
+## Two things not to lose
+
+**The freemium promotional campaign does not come with this code.** OpenOutreach shipped a campaign
+advertising itself, sent from the operator's own mailbox. It has been reclassified from growth asset
+to install tax and dies with the sender — it should not reappear here. `LEGAL_NOTICE.md` §4 in
+OpenOutreach is the disclosure that describes it.
+
+**The measured send cap is the good part.** `warmth.py` reads the box's own Sent folder and derives a
+daily ceiling from what that mailbox already does, rather than taking a configured number. It replaced
+a declared 50/day that nobody had measured. Whatever this becomes, keep the principle: the receiver
+counts every message the box emits, so the ledger has to count the same things.

cold_outreach/__main__.py

@@ -0,0 +1,172 @@
+"""`outsend` — the command on the right of the pipe.
+
+    openoutreach find 50 --json | outsend
+
+**On the pipe it takes no verb.** Things on the right of a pipe conventionally take
+none: `| less`, `| jq`, `| tee`. The one argument it has is the campaign, and that one
+is usually absent too. The two verbs it does have are the ones that are not on the pipe
+at all: `send`, which reads no stdin and mails what is already stored, and `init`,
+which asks for what a first run needs — what the campaign sells and to whom, who is
+signing the mail, and the mailbox it leaves from.
+
+**Ingesting and sending are separate invocations on purpose.** A pipe's right-hand side
+must not block on the network while a producer is still writing, and the cadence the two
+want is different — leads arrive when `find` runs, mail moves on the mailbox's clock. So
+the cron line is two entries, not one command doing both.
+
+**stdout is reserved and stays clean**, so the stream composes and a receipt can
+never corrupt it. Everything narrated — the campaign resolved, the counts, a skipped
+line, Django's own migration chatter — goes to stderr.
+
+**The exit code is the only acknowledgement a pipe can carry**: 0 means the rows are
+durably persisted. Non-zero after a skipped line does *not* mean *nothing to
+reconcile* — the good rows are stored and a re-run is safe, because ingest is
+idempotent; it means a producer emitted something unreadable and somebody has to know.
+"""
+from __future__ import annotations
+
+import argparse
+import logging
+import os
+import sys
+
+USAGE = """outsend [--campaign NAME]        read JSON Lines on stdin, store them, exit
+outsend send [--campaign NAME]   read the mail, answer replies, open what the guards allow
+outsend init [--campaign NAME]   collect what a first run needs — the campaign, who you
+                                 are, and a mailbox to send from"""
+
+
+def main(argv: list[str] | None = None) -> int:
+    """Entry point for the `outsend` console script. Returns the exit code."""
+    args = _parse_args(argv if argv is not None else sys.argv[1:])
+    _configure_logging(args.debug)
+    _boot()
+
+    from cold_outreach.errors import OutsendError
+
+    try:
+        return {"init": _init, "send": _send}.get(args.command, _ingest)(args)
+    except OutsendError as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return 1
+
+
+def _parse_args(argv: list[str]) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(prog="outsend", usage=USAGE)
+    parser.add_argument("command", nargs="?", choices=["init", "send"])
+    parser.add_argument("--campaign", default=None,
+                        help="which campaign these leads belong to; required only if there are several")
+    parser.add_argument("--debug", action="store_true", help="log what each step decided")
+    return parser.parse_args(argv)
+
+
+def _configure_logging(debug: bool) -> None:
+    """Everything to stderr, because stdout belongs to the pipe."""
+    logging.basicConfig(
+        stream=sys.stderr,
+        level=logging.DEBUG if debug else logging.INFO,
+        format="%(message)s",
+    )
+
+
+def _boot() -> None:
+    """Start Django and bring the store up to date.
+
+    Migrations run on every invocation and are a no-op once applied, so a fresh
+    install is an ingest that works rather than a traceback about a missing table.
+    Their narration is pointed at stderr for the same reason everything else is; a
+    migration that *fails* still raises, because that is a broken install and not an
+    answer.
+    """
+    import django
+
+    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "cold_outreach.settings")
+    django.setup()
+
+    from django.core.management import call_command
+
+    call_command("migrate", verbosity=0, stdout=sys.stderr)
+
+
+# ── The pipe ──────────────────────────────────────────────────────
+
+
+def _campaign_for(args: argparse.Namespace):
+    """The campaign this invocation acts on, hydrated from the environment and narrated.
+
+    Every command starts here, so `--campaign` means the same thing to all of them and
+    the operator reads which one was chosen before anything else is printed.
+    """
+    from cold_outreach.leads.campaigns import hydrate_from_environment, resolve_campaign
+
+    campaign = resolve_campaign(args.campaign)
+    hydrate_from_environment(campaign)
+    print(f"campaign: {campaign.name}", file=sys.stderr)
+    return campaign
+
+
+def _ingest(args: argparse.Namespace) -> int:
+    """Read stdin into the resolved campaign and report what happened."""
+    from cold_outreach.leads.ingest import ingest
+
+    result = ingest(sys.stdin, _campaign_for(args))
+    print(f"stored {result.stored} lead(s)", file=sys.stderr)
+    if result.suppressed:
+        print(f"{result.suppressed} of them are suppressed and will not be emailed", file=sys.stderr)
+    if result.skipped:
+        print(f"{result.skipped} line(s) skipped — see above; the rest are stored", file=sys.stderr)
+    return 0 if result.ok else 1
+
+
+# ── The send pass ─────────────────────────────────────────────────
+
+
+def _send(args: argparse.Namespace) -> int:
+    """One bounded pass: read the mail, answer what came back, open what fits.
+
+    **`init` runs implicitly here**, because a send is the first moment the campaign's
+    fields, the operator's name and a mailbox actually have to be there — and an operator
+    who wired the pipe into a timer should not discover a setup step they never ran. On a
+    TTY that is the same prompts; headless it is the same error naming the variables,
+    raised before any mail moves.
+
+    Exit code is the pass's own: non-zero when something failed on its way out, so a
+    timer's failure mail carries it.
+    """
+    from cold_outreach.first_run import ensure_ready
+    from cold_outreach.send_pass import run_send_pass
+
+    campaign = _campaign_for(args)
+    ensure_ready(campaign)
+
+    result = run_send_pass(campaign)
+    print(f"read {result.mirrored} new message(s) · answered {result.answered} · "
+          f"opened {result.opened}", file=sys.stderr)
+    if result.failed:
+        print(f"{result.failed} send(s) failed — see above; the next pass tries them again",
+              file=sys.stderr)
+    return 0 if result.ok else 1
+
+
+# ── First run ─────────────────────────────────────────────────────
+
+
+def _init(args: argparse.Namespace) -> int:
+    """Collect the three things a send needs, and say what it ended up with.
+
+    The collecting itself lives in `first_run.py`, because `send` does exactly the same
+    thing before any mail moves and the two must not drift.
+    """
+    from cold_outreach.core.operator import seller_full_name
+    from cold_outreach.emails.models import Mailbox
+    from cold_outreach.first_run import ensure_ready
+
+    campaign = _campaign_for(args)
+    ensure_ready(campaign)
+    print(f"campaign {campaign.name} is ready: signed by {seller_full_name()}, "
+          f"sending from {Mailbox.objects.first()}", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

cold_outreach/core/agents/outreach.py

@@ -0,0 +1,190 @@
+# cold_outreach/core/agents/outreach.py
+"""The outreach agent: one prompt, one decision type, both ends of the thread.
+
+There is a single conversational agent. The cold open and every later reply are
+the same voice doing the same job — Mom Test research, not selling — so they
+render from one template (``outreach_agent.j2``) which branches on the only thing
+that actually differs: whether a thread exists yet.
+
+- **first touch** (no thread): the agent must ``send_message`` and must supply a
+  ``subject``. ``emails/steps/send.py`` sends it and records the thread root.
+- **in thread**: the agent only ever runs on a thread the lead has *replied* to —
+  the mail pass has already written the reply — so it picks ``send_message`` /
+  ``mark_completed`` / ``suppress``. ``emails/steps/reply.py`` executes the choice.
+
+There is no ``wait`` and no follow-up interval, because nobody is chased: silence
+is not a decision the agent gets to make, it is simply the absence of work.
+
+``suppress`` is the *worded* unsubscribe — "take me off your list", "stop
+emailing me". It threads like any other reply, so the box-wide alias scan in
+``emails/classify.py`` can never see it, and the agent reading every reply already
+can. It is a stronger statement than ``not_interested``: it ends the thread and
+suppresses the person account-wide, across every campaign.
+
+Single LLM call with structured output — no tool-calling loop.
+"""
+from __future__ import annotations
+
+import logging
+from datetime import datetime, timedelta
+from typing import Literal
+
+from django.utils import timezone
+from pydantic import BaseModel, Field, model_validator
+from pydantic_ai import Agent
+
+from cold_outreach.core.agents.prompt import _format_facts, base_context, render
+from cold_outreach.core.business_time import business_days_between
+from cold_outreach.core.llm import get_llm_model, run_agent_sync
+
+logger = logging.getLogger(__name__)
+
+
+class OutreachDecision(BaseModel):
+    """Structured output from the outreach agent, at either end of the thread."""
+
+    action: Literal["send_message", "mark_completed", "suppress"] = Field(
+        description=(
+            "What to do next for this lead. The first email in a thread is always send_message. "
+            "Use suppress when the lead asked to stop being contacted."
+        ),
+    )
+    subject: str | None = Field(
+        default=None,
+        description=(
+            "Subject line. Required on the first email of a thread — short and specific, "
+            "like a real person wrote it, not salesy. Omit when replying in an existing thread."
+        ),
+    )
+    message: str | None = Field(
+        default=None,
+        description="The email to send. Required when action='send_message'. A few short sentences, no signature.",
+    )
+    outcome: Literal[
+        "converted", "not_interested", "wrong_fit", "no_budget",
+        "has_solution", "bad_timing", "unresponsive",
+    ] | None = Field(
+        default=None,
+        description="Why the conversation ended. Required when action='mark_completed'.",
+    )
+    @model_validator(mode="after")
+    def _check_required_fields(self):
+        if self.action == "send_message" and not self.message:
+            raise ValueError("message is required when action='send_message'")
+        if self.action == "mark_completed" and not self.outcome:
+            raise ValueError("outcome is required when action='mark_completed'")
+        return self
+
+
+# Number of trailing verbatim messages the agent sees alongside the rolling
+# chat_summary. Older turns live in the summary fact list; the recency window
+# preserves literal phrasing for the turns that matter most when composing
+# the next reply.
+RECENT_MESSAGES_WINDOW = 6
+
+
+def run_outreach_agent(deal) -> OutreachDecision:
+    """Decide the next move for ``deal`` — the cold open or the answer to a reply.
+
+    The caller has already folded any new inbound messages into ``deal.chat_summary``
+    (``emails/steps/reply.py``), so this reads the conversation store rather than the
+    mailbox — no IMAP here. On a first touch there is nothing to read, and the
+    decision is validated to be a sendable opener with a subject.
+    """
+    public_id = deal.lead.public_id
+    is_first_touch = not deal.thread_id
+
+    recent = [] if is_first_touch else _load_recent_messages(deal)
+    system_prompt = _render_system_prompt(deal, recent, is_first_touch)
+
+    agent = Agent(
+        get_llm_model(),
+        output_type=OutreachDecision,
+        model_settings={"temperature": 0.7, "timeout": 60},
+    )
+    decision = run_agent_sync(agent.run(system_prompt)).output
+    if decision is None:
+        raise RuntimeError(f"LLM returned unparseable response for outreach to {public_id}")
+    if is_first_touch:
+        _validate_opener(decision, public_id)
+
+    logger.info("outreach agent for %s: %s", public_id, decision.action)
+    return decision
+
+
+def _validate_opener(decision: OutreachDecision, public_id: str) -> None:
+    """A first touch must be a sendable email with its own subject."""
+    if decision.action != "send_message":
+        raise ValueError(f"opener for {public_id} must send_message, got {decision.action}")
+    if not decision.subject:
+        raise ValueError(f"opener for {public_id} has no subject")
+
+
+# ── Prompt context ────────────────────────────────────────────────
+
+
+def _render_system_prompt(deal, recent_messages: list, is_first_touch: bool) -> str:
+    """Render the outreach prompt for whichever end of the thread we're at."""
+    now = timezone.now()
+    thread_context = {} if is_first_touch else {
+        "chat_summary": _format_facts(deal.chat_summary),
+        "recent_messages": _format_recent_messages(recent_messages, now),
+        "today": now.strftime("%Y-%m-%d"),
+        "business_days_since_last_outgoing": _business_days_since_last_outgoing(recent_messages, now),
+    }
+    return render(
+        "outreach_agent.j2",
+        **base_context(deal),
+        is_first_touch=is_first_touch,
+        **thread_context,
+    )
+
+
+def _load_recent_messages(deal, limit: int = RECENT_MESSAGES_WINDOW) -> list:
+    """The thread's last `limit` **turns**, in chronological order.
+
+    The recency window of verbatim turns the agent sees alongside the rolling
+    ``chat_summary`` — the opener plus any replies read from the mailbox. Turns
+    only, so a non-delivery report cannot reach the agent's reasoning and be
+    answered as though a person had written it.
+    """
+    if not deal.thread_id:
+        return []
+    return list(reversed(list(deal.thread.turns().order_by("-sent_at", "-pk")[:limit])))
+
+
+def _format_recent_messages(messages: list, now: datetime) -> str:
+    """Render the last few turns as a timestamped transcript."""
+    if not messages:
+        return "No recent messages."
+    lines = []
+    for m in messages:
+        content = (m.body_text or "").strip()
+        if not content:
+            continue
+        speaker = "Me" if m.is_outbound else "Lead"
+        prefix = f"{speaker} ({_humanize_age(m.sent_at, now)})" if m.sent_at else speaker
+        lines.append(f"{prefix}: {content}")
+    return "\n".join(lines) or "No recent messages."
+
+
+def _humanize_age(when: datetime, now: datetime) -> str:
+    """Render `when` as a coarse age relative to `now` (e.g. ``3d ago``)."""
+    delta = now - when
+    if delta < timedelta(hours=1):
+        return f"{max(int(delta.total_seconds() // 60), 1)}m ago"
+    if delta < timedelta(days=1):
+        return f"{int(delta.total_seconds() // 3600)}h ago"
+    return f"{delta.days}d ago"
+
+
+def _business_days_since_last_outgoing(messages: list, now: datetime) -> int | None:
+    """Whole working days since the most recent outgoing message, or None if there are none.
+
+    Weekends don't count — a Friday send read on Monday is one working day old,
+    which is the gap the agent should pace against.
+    """
+    timestamps = [m.sent_at for m in messages if m.is_outbound and m.sent_at]
+    if not timestamps:
+        return None
+    return business_days_between(max(timestamps), now)

cold_outreach/core/agents/prompt.py

@@ -0,0 +1,57 @@
+# cold_outreach/core/agents/prompt.py
+"""Jinja plumbing + the shared context for the outreach prompt.
+
+One template renders both the cold open and every in-thread reply
+(``outreach_agent.j2``); ``base_context`` is the half of its variables that
+doesn't depend on where in the thread we are. ``core.agents.outreach`` adds the
+conversation half.
+"""
+from __future__ import annotations
+
+import jinja2
+
+from cold_outreach.core.conf import PROMPTS_DIR
+from cold_outreach.core.operator import seller_full_name
+from cold_outreach.leads.summaries import facts_of
+
+_ENV = jinja2.Environment(loader=jinja2.FileSystemLoader(str(PROMPTS_DIR)))
+
+
+def render(template_name: str, **context) -> str:
+    """Render a prompt template by name from the shared prompts dir."""
+    return _ENV.get_template(template_name).render(**context)
+
+
+def base_context(deal) -> dict:
+    """The channel-agnostic prompt variables shared by every outreach entrypoint.
+
+    Four of the five come from the campaign and the operator. The fifth is what we
+    know about the person: the facts extracted from the raw ``profile_text`` the pipe
+    carried (``leads/summaries.py``), falling back to that text itself if the
+    extraction has not run — the same information either way, and a lead is never
+    described to the agent as *(none yet)* while the sentences are sitting in the row.
+    """
+    campaign = deal.campaign
+    return {
+        "self_name": seller_full_name(),
+        "product_docs": campaign.product_docs or "",
+        "campaign_target": campaign.campaign_target or "",
+        "booking_link": campaign.booking_link or "",
+        "profile_summary": _format_profile(deal.lead),
+    }
+
+
+def _format_profile(lead) -> str:
+    """The lead's facts as bullets, or the raw profile text, or an admission of nothing."""
+    facts = facts_of(lead.profile_summary)
+    if facts:
+        return "\n".join(f"- {fact}" for fact in facts)
+    return lead.profile_text or "(nothing on file)"
+
+
+def _format_facts(summary: dict | None) -> str:
+    """Render a ``{"facts": [...]}`` blob as a bullet list."""
+    facts = facts_of(summary)
+    if not facts:
+        return "(none yet)"
+    return "\n".join(f"- {fact}" for fact in facts)

cold_outreach/core/business_time.py

@@ -0,0 +1,47 @@
+# cold_outreach/core/business_time.py
+"""Working-day arithmetic — how old a thread really is.
+
+A thread that sat untouched over a weekend has not really gone quiet, and the
+outreach agent is told the gap in working days so it reads a Friday reply answered
+on Monday as one day old rather than three.
+
+Measuring is all this does. Public holidays are deliberately not modelled: they are
+per-country, per-year data we do not carry, and this figure is coarse enough that a
+missed holiday costs one slightly-off number in a prompt.
+"""
+from __future__ import annotations
+
+from datetime import date, datetime, timedelta
+
+SATURDAY = 5
+
+
+def is_business_day(day: date) -> bool:
+    """True for Monday–Friday."""
+    return day.weekday() < SATURDAY
+
+
+def business_days_between(start: datetime, end: datetime) -> int:
+    """Whole working days elapsed from ``start`` to ``end`` (weekends excluded).
+
+    Counts the business days in the half-open range ``(start.date(), end.date()]``,
+    so Friday → Monday is 1 and Friday → Sunday is 0. Never negative.
+    """
+    return _business_days_in_range(start.date() + timedelta(days=1),
+                                   end.date() + timedelta(days=1))
+
+
+# ── Internals ─────────────────────────────────────────────────────
+
+
+def _business_days_in_range(start: date, end: date) -> int:
+    """Number of Monday–Friday days in the half-open range ``[start, end)``."""
+    days = (end - start).days
+    if days <= 0:
+        return 0
+    full_weeks, remainder = divmod(days, 7)
+    count = full_weeks * 5
+    for offset in range(remainder):
+        if is_business_day(start + timedelta(days=offset)):
+            count += 1
+    return count