logquill 0.2.0 → 0.4.0

package/README.md

@@ -10,22 +10,44 @@ JSON log shape with its Python sibling, [`logquill`](https://pypi.org/project/lo
 (repo: `logquill-python`).
 
 Status: pre-release, under active development. The core `Logger`, level
-filtering, the plugin pipeline (with `ContextPlugin`/`RedactPlugin`/
-`SamplingPlugin` built in), `JSONFormatter`, and the built-in transports are
-implemented; non-blocking async dispatch is not yet — see `CHANGELOG.md`
-for what's landed so far.
+filtering, a full plugin pipeline (context/redaction/PII/tamper-evidence/
+sampling/alerting), `JSONFormatter`, and a broad transport catalog —
+console/file/HTTP, SQL, NoSQL, message queues, and cloud-native log
+platforms — are implemented; non-blocking async dispatch is not yet — see
+`CHANGELOG.md` for what's landed so far.
 
 ## Features
 
 - **Structured by default** — every call carries a `meta` object, not just a message string
 - **Cross-language record shape** — identical JSON shape and level names/weights as [`logquill` on PyPI](https://pypi.org/project/logquill/)
 - **Pluggable formatters** — `JSONFormatter` out of the box; implement `format(record) -> string` for your own
-- **Pluggable transports** — `ConsoleTransport` (colorized, respects `NO_COLOR`, isomorphic), `FileTransport` (rotation), `HTTPTransport` (batched, `fetch`-based); write your own by subclassing `Transport`; `CollectingTransport` ships as an in-memory sink, handy for tests
-- **Plugin pipeline** — `ContextPlugin`, `RedactPlugin`, `SamplingPlugin` out of the box; `beforeLog`/`afterLog`/`onError` hooks; a throwing plugin can't crash logging
+- **A transport for wherever your logs need to go** — console, file, and HTTP out of the box, plus SQL (SQLite/Postgres/MySQL), NoSQL (MongoDB/DynamoDB/Redis), message queues (Kafka/RabbitMQ/SQS/Pub-Sub), and cloud-native platforms (CloudWatch/Cloud Logging/App Insights/Datadog/Elasticsearch/New Relic) — see [Transports](#transports). Every backend driver is an **optional peer dependency**: install only the one you use, or inject a pre-built client. Write your own by subclassing `Transport`; `CollectingTransport` ships as an in-memory sink, handy for tests
+- **Plugin pipeline** — `ContextPlugin`, `RedactPlugin`, `PIIRedactPlugin`, tail-based `SamplingPlugin`, `TamperEvidentPlugin`, and `AlertingPlugin` (`SlackAlertPlugin`/`PagerDutyAlertPlugin`/`EmailAlertPlugin`) out of the box; `beforeLog`/`afterLog`/`onError` hooks, or just pass a plain function to `.use()`; a throwing plugin can't crash logging — see [Plugins](#plugins)
 - **Child loggers** — `.child()` inherits level, transports, and plugins, and merges its own `meta` on top
 - **Typed throughout** — TypeScript strict mode, no `any` in the public API
 - **Dual package** — works via both `require()` (CJS) and `import` (ESM) from the same published package
-- *(planned)* non-blocking async dispatch, `AsyncLocalStorage`-based context propagation — see `CHANGELOG.md`
+- **Tracing & agentic logging** — `.thought()/.action()/.observation()/.decision()`, `Logger.span()` for nested/durationMs-stamped spans, `RunPlugin` (per-run id + step counter), and `TraceContextPlugin` (cross-service `traceId`, OTel-aware) — see [Tracing & agentic logging](#tracing--agentic-logging)
+- **LangChain.js / LangGraph.js adapter** — `LangChainAdapter`, a `BaseCallbackHandler` that maps chain/LLM/tool/agent events onto the calls above with zero manual instrumentation, from a separate `logquill/langchain` entry point — see [Agentic framework adapters](#agentic-framework-adapters)
+- *(planned)* non-blocking async dispatch, `AsyncLocalStorage`-based general context propagation — see `CHANGELOG.md`
+
+## Contents
+
+- [Install](#install)
+- [Usage](#usage)
+- [Transports](#transports)
+  - [SQL transports](#sql-transports)
+  - [NoSQL transports](#nosql-transports)
+  - [Message queue transports](#message-queue-transports)
+  - [Cloud-native transports](#cloud-native-transports)
+- [Plugins](#plugins)
+  - [Tail-based sampling](#tail-based-sampling)
+  - [PII redaction](#pii-redaction)
+  - [Tamper-evident logs](#tamper-evident-logs)
+  - [Alerting](#alerting)
+- [Tracing & agentic logging](#tracing--agentic-logging)
+- [Agentic framework adapters](#agentic-framework-adapters)
+- [Development](#development)
+- [License](#license)
 
 ## Install
 
@@ -90,7 +112,36 @@ a dual build with full TypeScript types.
 
 Attach transports to a `Logger` to actually write records somewhere. Each
 record is dispatched to every attached transport synchronously
-(non-blocking dispatch isn't implemented yet):
+(non-blocking dispatch isn't implemented yet). Every backend driver below
+is an **optional peer dependency** — `npm install` only pulls in what you
+actually import; nothing is installed on your behalf.
+
+| Transport | Backend | Peer dependency | Setup |
+|---|---|---|---|
+| `ConsoleTransport` | stdout/stderr via `console.*` | *(none)* | zero-setup, isomorphic |
+| `FileTransport` | local file, with rotation | *(none)* | zero-setup |
+| `HTTPTransport` | any HTTP log endpoint | *(none, uses `fetch`)* | zero-setup |
+| `SQLiteTransport` | SQLite | `better-sqlite3` | zero-setup (file or `:memory:`) |
+| `PostgresTransport` | PostgreSQL | `pg` | needs a server |
+| `MySQLTransport` | MySQL | `mysql2` | needs a server |
+| `MongoDBTransport` | MongoDB | `mongodb` | needs a server |
+| `DynamoDBTransport` | AWS DynamoDB | `@aws-sdk/client-dynamodb` | needs an AWS account |
+| `RedisTransport` | Redis Streams | `redis` | needs a server |
+| `KafkaTransport` | Kafka | `kafkajs` | needs a broker |
+| `RabbitMQTransport` | RabbitMQ | `amqplib` | needs a broker |
+| `SQSTransport` | AWS SQS | `@aws-sdk/client-sqs` | needs an AWS account |
+| `PubSubTransport` | GCP Pub/Sub | `@google-cloud/pubsub` | needs a GCP account |
+| `CloudWatchTransport` | AWS CloudWatch Logs | `@aws-sdk/client-cloudwatch-logs` | needs an AWS account |
+| `CloudLoggingTransport` | GCP Cloud Logging | `@google-cloud/logging` | needs a GCP account |
+| `AppInsightsTransport` | Azure Application Insights | `applicationinsights` | needs an Azure account |
+| `DatadogTransport` | Datadog Logs | *(none, uses `fetch`)* | needs a Datadog account |
+| `ElasticsearchTransport` | Elasticsearch `_bulk` API | *(none, uses `fetch`)* | needs a cluster |
+| `NewRelicTransport` | New Relic Log API | *(none, uses `fetch`)* | needs a New Relic account |
+
+Every transport also accepts an injected client/sender in place of its
+default driver — the pattern every example below and every transport's own
+test suite uses, so you never need a live backend just to test your
+logging setup.
 
 ```ts
 import { ConsoleTransport, FileTransport, HTTPTransport, Logger } from "logquill";
@@ -366,7 +417,209 @@ logger.info("login attempt", { user_id: 42, password: "hunter2" });
 // (unless this call was one of the ~90% sampling dropped, in which case it's null)
 ```
 
-Write your own by implementing `Plugin`; every hook is optional.
+Write your own by implementing `Plugin`; every hook is optional. `.use()`
+also accepts a plain function in place of a `Plugin` — sugar for a
+single-method `beforeLog` plugin, Express/Koa-style:
+
+```ts
+logger.use((record) => {
+  delete record.meta.ssn;
+  return record; // or null to drop the record
+});
+```
+
+### Tail-based sampling
+
+`SamplingPlugin` can do more than flat-rate sampling: pass `transports`
+and it buffers a sampled-out record under its `meta.traceId` instead of
+dropping it outright. If a later record on that same trace reaches
+`elevateAt` (default `ERROR`), the whole trace is flushed — every buffered
+record for it, plus everything from then on — so a request that turned out
+to matter still produces a complete trace, even though most of its steps
+would otherwise have been sampled away.
+
+```ts
+import { CollectingTransport, Logger, SamplingPlugin } from "logquill";
+
+const sink = new CollectingTransport();
+const logger = new Logger("app", {
+  transports: [sink],
+  plugins: [new SamplingPlugin(0.01, { transports: [sink] })], // keep 1%, but never lose an errored trace
+});
+
+logger.info("step 1", { traceId: "req-42" }); // likely dropped...
+logger.info("step 2", { traceId: "req-42" }); // ...and this one too
+logger.error("step 3 failed", { traceId: "req-42" }); // elevates req-42 — steps 1-3 all ship
+```
+
+### PII redaction
+
+`PIIRedactPlugin` complements `RedactPlugin`'s exact-key matching with
+regex-based scanning of `meta` **values** — emails, SSNs, credit-card
+numbers, and phone numbers are redacted wherever they appear, recursively
+through nested objects/arrays, regardless of which key holds them.
+
+```ts
+import { Logger, PIIRedactPlugin } from "logquill";
+
+const logger = new Logger("app", { plugins: [new PIIRedactPlugin()] });
+logger.info("support ticket", { notes: "contact me at jane@example.com" });
+// meta.notes: "contact me at ***"
+```
+
+### Tamper-evident logs
+
+`TamperEvidentPlugin` hash-chains every record — each one's `meta.hash` is
+a SHA-256 digest over its own content plus the previous record's hash — so
+editing, removing, or reordering a written line breaks the chain from that
+point on, detectable later with the static `verifyChain()`. Opt-in: hashing
+every record has a real CPU cost.
+
+```ts
+import { Logger, TamperEvidentPlugin } from "logquill";
+
+const logger = new Logger("app", { plugins: [new TamperEvidentPlugin()] });
+const records = [logger.info("one"), logger.info("two")];
+
+TamperEvidentPlugin.verifyChain(records); // true
+```
+
+### Alerting
+
+`AlertingPlugin` is the base for plugins that fire an external alert on
+ERROR/FATAL (or any configurable `threshold`) without ever blocking the
+log call that triggered it. Repeated matches within a dedupe window
+collapse into a single follow-up alert reporting the total count, instead
+of spamming the destination once per record.
+
+```ts
+import { Logger, SlackAlertPlugin } from "logquill";
+
+const logger = new Logger("app", {
+  plugins: [new SlackAlertPlugin("https://hooks.slack.com/services/...")],
+});
+logger.error("payment webhook failed", { orderId: "o-123" });
+```
+
+`PagerDutyAlertPlugin` (Events API v2, no extra dependency) and
+`EmailAlertPlugin` (SMTP via the optional `nodemailer` peer dependency, or
+inject a `sender`) follow the same shape. Write your own by extending
+`AlertingPlugin` and implementing `sendAlert(record, occurrences)`.
+
+## Tracing & agentic logging
+
+`.thought()/.action()/.observation()/.decision()` are `.info()` with
+`meta.kind` pre-set, for tagging steps of an agent's reasoning loop:
+
+```ts
+import { Logger } from "logquill";
+
+const logger = new Logger("agent");
+logger.thought("deciding which tool to call", { candidates: ["search", "calculator"] });
+logger.action("calling search", { query: "current weather in nyc" });
+logger.observation("search returned 3 results");
+logger.decision("using search result #1");
+```
+
+`Logger.span()` wraps an operation, emitting one record on completion with
+`meta.spanId`/`meta.durationMs` — every record logged inside it, including
+across an `await`, is automatically stamped with `meta.parentSpanId`, so a
+full run's nesting can be reconstructed by sorting on `spanId`/`parentSpanId`.
+It still emits (at `ERROR`, with `meta.error`) and rethrows if the block
+throws:
+
+```ts
+import { Logger } from "logquill";
+
+async function callLlm(prompt: string): Promise<string> {
+  return `response to: ${prompt}`;
+}
+
+const logger = new Logger("agent");
+
+const answer = await logger.span(
+  "callLlm",
+  async () => {
+    logger.action("requesting completion", { model: "gpt-4" });
+    const response = await callLlm("current weather in nyc");
+    logger.observation("received completion");
+    return response;
+  },
+  { model: "gpt-4" },
+);
+```
+
+`RunPlugin` stamps `meta.runId` (generated, or given explicitly) plus an
+incrementing `meta.step` — attach a fresh instance per run so concurrent
+runs don't share a counter:
+
+```ts
+import { Logger, RunPlugin } from "logquill";
+
+const runLogger = new Logger("app").child("agent").use(new RunPlugin());
+runLogger.thought("step one"); // meta: { kind: "thought", runId: "...", step: 0 }
+runLogger.action("step two");  // meta: { kind: "action", runId: "...", step: 1 }
+```
+
+`TraceContextPlugin` stamps `meta.traceId`, for correlating one request
+across services — distinct from `runId`, which scopes one agent run. It
+resolves, in priority order: an active OpenTelemetry span (if
+`@opentelemetry/api` is installed — never a required dependency), an
+inbound `traceparent`/X-Ray/GCP trace header, or a freshly generated id:
+
+```ts
+import { Logger, setTraceparent, TraceContextPlugin } from "logquill";
+
+const logger = new Logger("app", { plugins: [new TraceContextPlugin()] });
+
+// in HTTP middleware, before the handler runs:
+const reset = setTraceparent(req.headers["traceparent"]);
+try {
+  logger.info("handling request"); // meta.traceId resolved from the inbound header
+} finally {
+  reset();
+}
+```
+
+## Agentic framework adapters
+
+`LangChainAdapter` implements LangChain.js's `BaseCallbackHandler`, mapping
+chain/LLM/tool/agent events onto `.action()/.observation()/.decision()` and
+`span()`-shaped records — pass it into `callbacks: [...]` and a chain's
+full call tree is captured with zero manual instrumentation:
+
+```ts
+import { RunnableLambda } from "@langchain/core/runnables";
+import { Logger, RunPlugin } from "logquill";
+import { LangChainAdapter } from "logquill/langchain";
+
+const logger = new Logger("agent").use(new RunPlugin());
+const handler = new LangChainAdapter(logger);
+
+const answerQuestion = RunnableLambda.from((question: string) => `answer: ${question}`);
+await answerQuestion.invoke("what is 2+2?", { callbacks: [handler] });
+// logs one record: { message: "RunnableLambda", meta: { kind: "span", spanId: "...", durationMs: ... } }
+```
+
+`LangGraphAdapter` is the same handler under its own name, for LangGraph.js
+graphs — its nodes run as ordinary LangChain `Runnable`s, so no extra
+mapping is needed; pass it the same way:
+
+```ts
+import { LangGraphAdapter } from "logquill/langchain";
+
+const handler = new LangGraphAdapter(logger);
+// const graph = builder.compile({ checkpointer });
+// await graph.invoke(input, { callbacks: [handler], configurable: { thread_id: "1" } });
+```
+
+This is a **separate entry point** — `import ... from "logquill/langchain"`,
+not the main `"logquill"` import — because `LangChainAdapter` has to
+`extends BaseCallbackHandler`, LangChain's own class. Importing plain
+`logquill` never touches `@langchain/core`; only importing
+`logquill/langchain` does. Install `@langchain/core` yourself (it's an
+optional peer dependency) — no separate `@langchain/langgraph` dependency
+is needed for `LangGraphAdapter`.
 
 ## Development