cognee 1.6.4.dev0__tar.gz → 1.6.5.dev0__tar.gz

cognee-1.6.5.dev0/.agents/skills/cognee-cli/SKILL.md

@@ -0,0 +1,161 @@
+---
+name: cognee-cli
+description: Use when the user wants to drive cognee from the terminal with cognee-cli — remember/recall/forget/improve memory commands, managing datasets and config, or database migrations.
+---
+
+# Use the cognee CLI
+
+`cognee-cli` ships with the package (entry point in `cognee/cli/_cognee.py`;
+each command lives in `cognee/cli/commands/`). Every command has
+`--help` for its flags, but only a few (`demo`, `eval`, `serve`,
+`push`, `upgrade`, `downgrade`, `stamp`, and `search` with one CODE example)
+include usage examples — for the memory commands use the examples
+in this file. Needs `LLM_API_KEY` configured, same as the SDK.
+
+## Core flow
+
+The memory commands are the primary surface as of cognee 1.x:
+
+```bash
+cognee-cli remember "Your text here"         # also accepts file paths / URLs
+cognee-cli remember ./docs --dataset-name my_project
+cognee-cli recall "Your question"            # query the graph
+cognee-cli recall "keyword" --query-type CHUNKS
+cognee-cli forget --all                      # wipe local state
+```
+
+`remember` is ingest + graph build in one step (`add` + `cognify` under the
+hood); `--background`/`-b` runs the cognify stage in the background, and
+`--dry-run` estimates LLM tokens/cost without ingesting. `recall` takes
+`--datasets`/`-d`, `--top-k`/`-k` (default 10), and `--session-id`/`-s`.
+
+`forget` targets `--dataset`, `--dataset-id`, `--data-id` (needs a dataset), or
+`--everything`/`--all` — one unified command replacing the older `delete` and
+empty-dataset paths. `--memory-only` (with a dataset) drops the graph and
+vectors but keeps the raw files, so the data can be rebuilt.
+
+> **`forget --all` does not ask for confirmation.** It deletes every dataset
+> immediately, even on a non-interactive stdin. The legacy `delete --all`
+> prompts `Delete ALL data from cognee? [y/N]` first, so switching to `forget`
+> silently drops that safety net — script it with care.
+
+`--query-type` accepts 10 of the SDK's 20 `SearchType` values — the list in
+`cognee/cli/config.py:SEARCH_TYPE_CHOICES`: HYBRID_COMPLETION, GRAPH_COMPLETION,
+RAG_COMPLETION, CHUNKS, CHUNKS_LEXICAL, SUMMARIES, CODE, CYPHER, GRAPH_REPORT,
+SKILLS. The rest (TEMPORAL, GRAPH_COMPLETION_COT,
+AGENTIC_COMPLETION, NATURAL_LANGUAGE, …) are SDK-only, e.g.
+`cognee.recall(q, query_type=SearchType.TEMPORAL)`.
+
+When `--query-type` is omitted the CLI uses `HYBRID_COMPLETION`
+(`DEFAULT_SEARCH_TYPE`), whereas the SDK's `cognee.recall()` auto-routes
+between search types. `--top-k` defaults to 10 on the CLI and 15 in the SDK.
+
+## Session memory and enrichment
+
+Session entries are currently written from the SDK — `cognee.remember(...,
+session_id="chat_1")` — not the CLI (`cognee-cli remember` has no session
+flag). The CLI side of session memory is reading and bridging:
+
+```bash
+cognee-cli recall "question" -s chat_1       # session cache first: without -d/-t
+                                             # this searches the session directly
+cognee-cli sessions get                      # retrieve session Q&A history
+cognee-cli improve -d my_project -s chat_1   # bridge session content into the graph
+cognee-cli improve -d my_project             # enrich/index the graph (no session)
+cognee-cli feedback ...                      # attach feedback to results
+```
+
+`improve` also takes `--feedback-alpha` (learning rate in
+(0, 1]; default `IMPROVE_FEEDBACK_ALPHA`, 0.1), `--build-global-context-index`,
+`--build-truth-subspace` (both opt-in stages; the truth subspace needs
+`-s`), and `--background`/`-b` (`--node-name` is deprecated: it warns and is
+ignored). It prints one line per stage — name, status
+(`completed` / `already_completed` / `skipped` / `errored`) and the skip
+reason (e.g. `no_session_ids`, `lock_held`, `opt_in_disabled`).
+`remember`/`improve` build their graphs through `cognify()`, so cognify-level
+settings (e.g. `PROVENANCE_TRACKING=true`) apply to them too.
+
+## Legacy / lower-level commands
+
+`add`, `cognify`, `search`, and `delete` still ship and are what the
+memory commands call underneath. Use them only to drive a single stage in
+isolation; prefer `remember`/`recall`/`forget`/`improve` otherwise.
+
+```bash
+cognee-cli add "text" && cognee-cli cognify  # what `remember` does in one step
+cognee-cli search "question"                 # `recall` minus routing/scope/session sources
+cognee-cli delete --all                      # superseded by `forget --all`
+```
+
+## Management
+
+```bash
+cognee-cli datasets list                     # dataset operations
+cognee-cli config get [key] [--show-secrets] # view one/all settings (API keys masked by default)
+cognee-cli config set <key> <value>          # set + persist to ./.env in the cwd
+cognee-cli config unset <key>                # reset a key to its default (also persisted)
+cognee-cli -ui                               # launch API server + UI (see cognee-server skill)
+cognee-cli serve --url http://localhost:8000 # connect CLI/SDK to a running instance
+```
+
+## Database migrations
+
+cognee has two migration chains: the relational schema (Alembic, in
+`cognee/alembic/`) and the graph/vector data chain (slugs registered in
+`cognee/modules/migrations/registry.py`). Both run automatically — at API
+server startup and on the first write (`remember`, `add`, `cognify`,
+`improve`, …) in an SDK/CLI process — unless `ENABLE_AUTO_MIGRATIONS=false`.
+So you rarely need these commands; they are for inspecting state, disabled
+auto-migration, and rollbacks. There is no `migrate` command.
+
+```bash
+cognee-cli current                    # stamped revision per database (per dataset
+                                      # with access control on)
+cognee-cli history                    # the data-migration chain, newest first
+cognee-cli upgrade                    # relational to head, then data chain to head
+cognee-cli upgrade <slug>             # data chain up to and including <slug>
+cognee-cli upgrade --alembic <rev>    # pin the relational (Alembic) target
+cognee-cli downgrade <slug|base>      # REWRITES DATA; revision is required,
+                                      # prompts unless --force; --dataset <uuid>
+                                      # (repeatable) limits it
+cognee-cli stamp <head|base|slug>     # set the stored revision WITHOUT running
+                                      # anything; prompts unless --force;
+                                      # --dataset <uuid> (repeatable) limits it
+```
+
+The positional revision is always a **data-chain slug**; the relational
+target goes through `--alembic`. `downgrade` leaves the relational schema
+alone unless you pass `--alembic`. `upgrade` runs even when
+`ENABLE_AUTO_MIGRATIONS=false`. `--alembic-path` (or `COGNEE_ALEMBIC_PATH`)
+points at a custom Alembic scripts directory.
+
+## Gotchas
+
+- The CLI initializes cognee lazily; the first command in a fresh environment
+  is slow (DB + model setup), later ones are fast.
+- `remember` (and `add`) without `--dataset-name` targets the default dataset
+  `main_dataset`; `recall`/`search` operate across your accessible datasets
+  unless a dataset is given.
+- `forget` refuses to run bare — pass `--dataset`, `--dataset-id`, `--data-id`
+  (with a dataset), or `--everything`/`--all`.
+- Session commands (`recall -s`, `sessions get`, `improve -s`) require
+  `CACHING=true` (the default) — with it off, session reads return nothing and
+  SDK session writes raise. To cut read latency and token cost while keeping
+  session memory, `cognee-cli config set AUTO_FEEDBACK false` — by default
+  cognee makes one structured-output LLM call per answered query to self-tune
+  its memory.
+- `config set`/`config unset` write to the `.env` file in whatever directory
+  you run the command from (creating it if missing). `config reset` (reset
+  *all* keys) is still not implemented.
+- **Which `.env` actually wins is not always the cwd one.** At import, cognee
+  calls `dotenv.load_dotenv(override=True)`, which resolves relative to the
+  *cognee package location*, not your working directory. In a source/editable
+  checkout (`uv pip install -e .`) a `.env` at the repo root therefore shadows
+  the `.env` in the directory you ran from — and because `override=True`, it
+  also beats variables you `export`ed. Symptom: `config set` appears to do
+  nothing, or the CLI connects to a backend you thought you had overridden.
+  To test against different settings, move the repo `.env` aside, or set
+  values programmatically after import (`cognee.config.set_*`). (Under
+  `python -c` the cwd `.env` does win, because dotenv falls back to the cwd
+  when `__main__` has no `__file__` — which is why the same command can
+  behave differently as a script vs. `-c`.)

cognee-1.6.5.dev0/.agents/skills/cognee-custom-graph-models/SKILL.md

@@ -0,0 +1,201 @@
+---
+name: cognee-custom-graph-models
+description: Use when defining the shape of cognee's knowledge graph with graph_model= — writing DataPoint node classes, choosing identity and index fields so nodes merge and are searchable, declaring typed Edge fields and FromIdentity references, building a model from a JSON schema, or debugging duplicated nodes, missing edges, or InvalidReferenceTypeError.
+---
+
+# Custom graph models
+
+By default cognee extracts a generic `KnowledgeGraph` of entities and
+relationships. Pass your own model with `graph_model=` and the LLM fills
+your node and edge types instead.
+
+```python
+from typing import Annotated, Literal
+import cognee
+from cognee.low_level import DataPoint, Edge, FromIdentity
+
+
+class Role(DataPoint):
+    name: str
+    metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
+
+
+class Person(DataPoint):
+    name: str
+    is_a: Annotated[Role, FromIdentity()] | None = None  # reference by name
+    reports_to: list[Edge["Person", "Person"]] = []  # edge owned by Person
+    metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
+
+
+class PeopleGraph(DataPoint):  # the root the LLM fills
+    people: list[Person]
+    friends_with: list[Edge[Person, Person]] = []
+    family: list[Edge[Person, Person, Literal["married_to", "sibling_of"]]] = []
+    metadata: dict = {"index_fields": [], "transparent": True}
+
+
+await cognee.remember(text, graph_model=PeopleGraph, custom_prompt="Extract every person...")
+```
+
+Full example: `examples/guides/custom_graph_model.py`.
+
+## Use it
+
+### Nodes: DataPoint classes
+
+Every node type subclasses `DataPoint` (`from cognee.low_level import
+DataPoint`). Its fields become:
+
+- **Node properties:** scalars, strings, dicts, and anything that is not a
+  DataPoint.
+- **Edges named after the field:** a field holding a DataPoint or a list of
+  DataPoints. `members: list[Person]` becomes `members` edges.
+
+`dict[str, Person]`, sets and plain tuples are stored as properties, not
+edges.
+
+### Identity and search: `metadata`
+
+| Key | What it does |
+|---|---|
+| `identity_fields` | The node id is derived from these field values (normalized: lowercased, spaces to `_`, apostrophes removed). The same entity from two chunks or two runs becomes **one node**. |
+| `index_fields` | Each field gets a vector collection named `<ClassName>_<field>`, so recall can find the node. |
+| `transparent` | The node is not stored; its children take its place. Use it for a root container like `PeopleGraph`. |
+
+**Without `identity_fields` every node gets a random id, so the same person
+is duplicated in every chunk and every run.** Set it on every node type that
+represents a real-world entity.
+
+**Write `metadata` explicitly**, as in the examples above. There is also an
+annotation shortcut (`from cognee.infrastructure.engine import Dedup,
+Embeddable`; `name: Annotated[str, Embeddable(), Dedup()]`), but today only
+half of it works:
+
+- `Dedup()` works: ids are derived from the marked fields.
+- `Embeddable()` does not index. The markers update the class-level
+  default, but each instance still carries `{"index_fields": []}`, and
+  indexing reads the instance, so no vector collection is created and
+  recall cannot find the node.
+
+Markers are also ignored entirely when the class declares `metadata`
+itself.
+
+### Typed edges: `list[Edge[Source, Target, Name]]`
+
+The LLM answers edges as flat rows of identity strings (`source`, `target`),
+and cognee resolves them to the extracted nodes. The third parameter
+controls the relationship name:
+
+| Declaration | Relationship name |
+|---|---|
+| `list[Edge[Person, Person]]` | The field name (`friends_with`) |
+| `list[Edge[Person, Person, Literal["a", "b"]]]` | The LLM picks one value |
+| `list[Edge[Person, Person, str]]` | Free-form from the LLM, normalized |
+
+- **Where to declare:** on the root model for relationships with no obvious
+  owner, or on the owning node. On the owner, endpoints of the owner's own
+  type must be strings (`Edge["Person", "Person"]`), because the class is
+  not defined yet inside its own body.
+- **Always a list:** `Edge[...]` or `Edge[...] | None` on its own raises.
+- **Both endpoint types need exactly one `identity_fields` entry.**
+
+### References: `Annotated[Target, FromIdentity()]`
+
+Instead of a nested object, the LLM answers the identity string of a node
+(`is_a: "engineer"`), and cognee links to that node. Supported spellings:
+`Target`, `Target | None`, `list[Target]`, `list[Target] | None`.
+Anything else raises `InvalidReferenceTypeError`. The target needs exactly
+one identity field, and its other required fields need defaults.
+
+### Edge values you build by hand
+
+`Edge(source=..., target=..., relationship_type=..., weight=...,
+properties={...})`. An omitted `source` falls back to the node declaring the
+field; on a parametrized field that node must be the declared `Source`
+type, or it raises. On a root container, always pass `source=`. The tuple
+form `(Edge(weight=0.8), target_node)` attaches edge properties to a plain
+DataPoint field (`examples/guides/custom_data_models.py`).
+
+### From JSON instead of Python
+
+- `cognee.low_level.graph_model_from_spec(spec)`: a small entity/relation
+  spec (names, fields, `one`/`many` relations) compiled to DataPoint
+  classes, with identity and index on `name` by default. Example:
+  `examples/guides/graph_model_from_json.py`.
+- `cognee.low_level.graph_schema_to_graph_model(json_schema)`: a JSON
+  Schema (needs a top-level `title`; only internal `#` refs).
+- HTTP: `POST /api/v1/remember` takes a `graph_model` form field (JSON
+  schema string); `POST /api/v1/cognify` takes a `graphModel` JSON object.
+  `POST /api/v1/llm/infer-schema` proposes a schema from sample text.
+
+Neither JSON path can express typed `Edge` fields or `FromIdentity`; use
+Python classes for those.
+
+## Pitfalls
+
+- **Duplicated nodes** → missing `identity_fields`.
+- **Node never shows up in recall** → no `index_fields`, or the recalling
+  process never imported the model class (graph completion searches the
+  collections of DataPoint classes loaded in that process).
+- **Edge rows silently missing** → an unresolved row is dropped with a
+  warning. Endpoints resolve by **exact type** against nodes extracted from
+  the same chunk: a subclass instance does not match where its base is
+  declared, and a node from another chunk or an earlier run is not a
+  candidate.
+- **`InvalidReferenceTypeError`, "declares Edge in a shape the LLM
+  extraction cannot fill"** → an unsupported `FromIdentity` or `Edge`
+  spelling. These are raised when the model is converted during
+  extraction, not at class definition, so they appear mid-pipeline.
+- **String endpoints resolve only to the owning model itself or a
+  module-level class.** A string naming another class defined inside a
+  function raises `InvalidReferenceTypeError`, so define models at module
+  level.
+- **Field names that collide with DataPoint's own fields** (`id`, `type`,
+  `version`, `metadata`, `created_at`, `belongs_to_set`, …) are stripped
+  from what the LLM sees. Rename them.
+- **A subclass that overrides `metadata`** replaces the parent's entirely.
+  Dropping `identity_fields` only logs a warning. A subclass also hashes ids
+  under its own class name.
+- **Write a `custom_prompt`.** Without one the generic knowledge-graph
+  prompt is used; your schema reaches the LLM only as structured output.
+- **Not with GLiNER.** `extractor="gliner_demo"` (alias `"gliner"`) raises
+  with a custom `graph_model`, and so does the default `GRAPH_EXTRACTOR=auto`
+  when no LLM key is configured (it resolves to `gliner_demo`).
+- **Remote mode drops it.** After `cognee.serve(url)`, `remember()` and
+  `cognify()` do not forward `graph_model`; the server builds a generic
+  graph.
+- A custom model skips the generic path's ontology resolution, per-graph
+  node dedup. Summaries still run.
+
+## How it works
+
+`extract_content_graph` converts a DataPoint model into a plain Pydantic
+model for the LLM: infrastructure fields and `metadata` are stripped, typed
+edge fields become row lists (`FriendsWithEdge` with `source`/`target`
+strings), and `FromIdentity` fields become strings. The answer is converted
+back into DataPoint instances with ids from `identity_fields`, edge rows are
+resolved against the nodes in that answer, and the result is attached to the
+chunk (`chunk.contains`) and stored by `add_data_points`. Ownership is
+recorded per document, so `forget(data_id=...)` removes a custom-model
+document's nodes while shared nodes survive.
+
+- LLM boundary, both directions: `cognee/shared/llm_graph_model.py`
+- DataPoint, metadata, ids: `cognee/infrastructure/engine/models/DataPoint.py`
+- Markers: `cognee/infrastructure/engine/models/FieldAnnotations.py`
+- Edge: `cognee/infrastructure/engine/models/Edge.py`
+- Property vs edge decision: `cognee/modules/graph/utils/field_edges.py`
+- Custom-model branch of extraction: `cognee/tasks/graph/extract_graph_from_data.py`
+- JSON schema / spec paths: `cognee/shared/graph_model_utils.py`,
+  `cognee/modules/graph_models/`
+- Vector collections: `cognee/tasks/storage/index_data_points.py`
+
+## Extending it
+
+- Tests for the LLM round trip: `cognee/tests/unit/modules/graph/test_content_graph_to_data_point.py`.
+  Edge typing: `cognee/tests/unit/interfaces/graph/test_typed_edge_model.py`,
+  `test_typed_edges_graph.py`. Identity: `cognee/tests/unit/infrastructure/engine/test_identity_fields.py`.
+  Property vs edge: `cognee/tests/unit/modules/graph/test_field_edges.py`.
+- Deletion of custom-model nodes: `cognee/tests/test_delete_custom_graph.py`.
+- A new `Edge` or `FromIdentity` spelling must be handled in both directions
+  in `llm_graph_model.py`, and rejected with `InvalidReferenceTypeError`
+  when unsupported, never silently accepted.

cognee-1.6.5.dev0/.agents/skills/cognee-custom-pipelines/SKILL.md

@@ -0,0 +1,219 @@
+---
+name: cognee-custom-pipelines
+description: Use when building your own cognee processing — writing custom tasks, chaining them into a pipeline with run_custom_pipeline or the lightweight run_pipeline (from cognee.pipelines import run_pipeline), storing custom DataPoints with add_data_points, running custom enrichment over the existing graph, checking pipeline run status, or debugging how data flows between tasks (batch_size, data_per_batch, ctx, Drop, enriches).
+---
+
+# Custom tasks and pipelines
+
+Everything cognee does runs as a **pipeline**: an ordered list of **tasks**,
+each a plain Python function whose output feeds the next one. `remember()`
+is the right tool for ordinary ingestion. Build a pipeline when you need
+processing cognee does not ship: your own extraction, your own node types,
+or a post-processing step over the graph.
+
+```python
+import cognee
+from cognee.modules.pipelines import Task
+from cognee.tasks.storage import add_data_points
+from cognee.low_level import DataPoint
+
+
+class Person(DataPoint):
+    name: str
+    metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
+
+
+async def extract_people(data_items: list) -> list[Person]:
+    people = []
+    for item in data_items:  # always a list, see below
+        text = item if isinstance(item, str) else ""
+        people += [Person(name=n.strip()) for n in text.split(",") if n.strip()]
+    return people
+
+
+result = await cognee.run_custom_pipeline(
+    tasks=[
+        Task(extract_people, needs_llm=False),
+        Task(add_data_points, needs_llm=False),  # store in graph + vector DBs
+    ],
+    data=["Ada Lovelace, Alan Turing"],
+    dataset="people",
+)
+```
+
+## Use it
+
+### Pick the runner
+
+There are three, and two share the name `run_pipeline`:
+
+| Runner | Import | Use it for |
+|---|---|---|
+| `cognee.run_custom_pipeline(...)` | `cognee` | The normal choice: runs your tasks against a dataset with permissions, a per-dataset lock, run records, and status |
+| Full orchestrator `run_pipeline(tasks=..., data=..., datasets=...)` | `cognee.modules.pipelines` | What `run_custom_pipeline` and `cognify` call; yields `PipelineRunInfo` |
+| Lightweight `run_pipeline([...], data=...)` | `from cognee.pipelines import run_pipeline` (after `import cognee`, the attribute `cognee.pipelines.run_pipeline` is the orchestrator) | Quick chains of `task()` specs with no permissions, locks, run rows, or migrations; returns the last step's outputs |
+
+`cognee.run_custom_pipeline(tasks, data=None, dataset="main_dataset",
+user=None, incremental_loading=False, data_per_batch=20,
+run_in_background=False, pipeline_name="custom_pipeline", data_cache=False,
+...)` returns `{dataset_id: PipelineRunInfo}` (the started run when
+`run_in_background=True`). With `data=None` it runs over the dataset's
+existing documents (`Data` rows).
+
+### Write a task
+
+```python
+from cognee.modules.pipelines import Task
+from cognee.modules.pipelines.models import PipelineContext
+from cognee.modules.pipelines.tasks.task import task_summary
+from cognee.pipelines import Drop
+
+
+@task_summary("Tagged {n} chunk(s)")
+async def tag_chunks(chunks: list, ctx: PipelineContext = None, label: str = "x"):
+    for chunk in chunks:
+        chunk.metadata["label"] = label
+    return chunks  # or yield per item; return/yield Drop to discard
+
+
+tag = Task(tag_chunks, label="reviewed", batch_size=10, needs_llm=False)
+```
+
+- A task is an `async def`, a generator, an async generator, or a plain
+  `def`. Extra `Task(fn, *args, **kwargs)` arguments are passed after the
+  pipeline data.
+- `needs_llm=False` on tasks that never call an LLM lets an LLM-free
+  pipeline skip the LLM connection check.
+- `ctx` (injected by the parameter **name** `ctx`) carries `user`,
+  `data_item`, `dataset`, `pipeline_run_id`, `pipeline_name`, and `extras`.
+- `task.with_config(batch_size=..., **kwargs)` returns a modified copy.
+
+### How data flows
+
+- **Each document runs the whole chain on its own** with `run_custom_pipeline`
+  or the orchestrator, and the first task receives it as a **one-element
+  list** (`[data_item]`), not the bare item. The lightweight `run_pipeline`
+  passes `data` to the first task unchanged.
+- **`data_per_batch`** (default 20) is how many documents run at the same
+  time. It is a concurrency limit, not a batch size.
+- **`batch_size` belongs to the consumer.** A task's `batch_size` decides how
+  the *previous* task's generator output is grouped before it is passed in.
+  Generator tasks always hand over lists; a coroutine or function hands over
+  its single return value.
+- **Streaming:** each upstream result goes down the chain immediately, so a
+  downstream task can run many times per document.
+- **`enriches=True`:** if the task returns `None`, its input is passed on
+  unchanged (coroutines and functions only, not generators).
+- **`Drop`:** returning or yielding it removes that item from the stream.
+- Every `DataPoint` passing through is stamped automatically with where it
+  came from (`source_pipeline`, `source_task`, `source_user`, …).
+
+### Store results
+
+`add_data_points(data_points, custom_edges=None, graph_only=False)` writes a list of `DataPoint`s to the graph and indexes
+their `index_fields` in the vector DB. It returns the same list, so it can
+sit mid-chain. Give every node type `identity_fields` so repeated runs merge
+instead of duplicating (see the `cognee-custom-graph-models` skill).
+
+### Work on the existing graph
+
+`cognee.memify`, `cognee-cli memify`, `POST /api/v1/memify` and `improve()`'s
+`extraction_tasks` / `enrichment_tasks` are deprecated and removed in the next
+release. Run a post-processing step over the
+graph with `run_custom_pipeline`: the task reads the graph itself, and one
+placeholder item as `data` makes the chain run once instead of once per
+document.
+
+```python
+from cognee.infrastructure.databases.graph import get_graph_engine
+
+
+async def enrich_graph(_items: list):
+    graph = await get_graph_engine()  # the dataset's graph inside the pipeline
+    nodes, edges = await graph.get_graph_data()
+    return [...]  # new DataPoints for add_data_points
+
+
+await cognee.run_custom_pipeline(
+    tasks=[Task(enrich_graph, needs_llm=False), Task(add_data_points, needs_llm=False)],
+    data=[{}],
+    dataset="people",
+)
+```
+
+For cognee's own enrichment (session bridging, feedback weights, conflict
+review, global context index) call `improve()`.
+
+### Check status
+
+```python
+status = await cognee.datasets.get_status([dataset_id], pipeline_names=["custom_pipeline"])
+```
+
+Without `pipeline_names` it reports only `cognify_pipeline`. It returns
+`{str(dataset_id): PipelineRunStatus}` (`{str(dataset_id): {pipeline_name:
+status}}` for several `pipeline_names`): `DATASET_PROCESSING_STARTED`,
+`_COMPLETED`, or `_ERRORED` (`_INITIATED` exists only on legacy rows). The value `run_custom_pipeline`
+returns per dataset is a `PipelineRunInfo` instead, whose class names the
+outcome: `PipelineRunCompleted`, `PipelineRunAlreadyCompleted`,
+`PipelineRunErrored`, and so on.
+
+## Pitfalls
+
+- **Wrong `run_pipeline`.** The one imported via
+  `from cognee.pipelines import run_pipeline` wants `task()`
+  specs *called* (`extract()`, not `extract`) and raises `TypeError`
+  otherwise; the orchestrator in `cognee.modules.pipelines` wants `Task`
+  objects and raises `WrongTaskTypeError` otherwise.
+- **`run_custom_pipeline` accepts only `Task` objects** despite its type hint;
+  string task names are not resolved.
+- **Some callables are rejected** by `Task` (`ValueError: Unsupported task
+  type`): bound methods and `functools.partial`s of a plain (non-generator)
+  sync function, and callable objects (instances with `__call__`). Generator
+  and async variants, plain functions, and lambdas work. When in doubt, wrap it
+  in a plain `def` / `async def`.
+- **`run_custom_pipeline` does not run database migrations.** On an existing
+  database, run `await cognee.run_migrations()` (or any `remember()` first).
+- **Nodes duplicate on every run** when a DataPoint has no
+  `identity_fields` (or `Dedup()` fields). `examples/guides/custom_data_models.py`
+  and `examples/guides/custom_tasks_and_pipelines.py` have this bug; don't
+  copy it.
+
+## How it works
+
+`run_custom_pipeline` → orchestrator `run_pipeline` (checks write
+permission, takes the per-dataset lock, records a `PipelineRun`) →
+`run_tasks` (a semaphore of `data_per_batch`, one chain per document) →
+`run_tasks_base` (streams each task's output into the next, batching by the
+consumer's `batch_size`, injecting `ctx`, stamping provenance).
+
+- Package overview and the runner semantics: `cognee/modules/pipelines/__init__.py`
+- `Task`, `task()`, `TaskSpec`, `BoundTask`, `@task_summary`: `cognee/modules/pipelines/tasks/task.py`
+- Orchestrator: `cognee/modules/pipelines/operations/pipeline.py`; execution:
+  `run_tasks.py`, `run_tasks_base.py`, `run_tasks_data_item.py`
+- Lightweight runner: `cognee/modules/pipelines/operations/run_pipeline.py`,
+  exported from `cognee/pipelines/`
+- Context: `cognee/modules/pipelines/models/PipelineContext.py`
+- `run_custom_pipeline`: `cognee/modules/run_custom_pipeline/run_custom_pipeline.py`
+- Storage: `cognee/tasks/storage/add_data_points.py`
+- Index of all shipped tasks: `cognee/tasks/README.md`
+
+Examples:
+
+- `examples/demos/custom_pipelines/custom_pipeline_single_object_example.py`:
+  the best reference. It runs over added documents, does LLM extraction into
+  typed DataPoints, then recalls. Its models declare identity with `Dedup()`
+  (the `Annotated` alternative to `metadata["identity_fields"]`).
+- `examples/demos/custom_pipelines/organizational_hierarchy/`: low-level
+  `run_tasks`, no LLM, dedup via `identity_fields`, status polling.
+- `examples/demos/custom_pipelines/custom_cognify_pipeline_example.py`:
+  rebuilds add + cognify from the default task list.
+
+## Extending it
+
+- **A new shipped task:** put it in the `cognee/tasks/` subpackage for its
+  stage, export it from that package's `__init__.py`, follow the template in
+  `cognee/tasks/README.md`, and add a unit test under `cognee/tests/unit/tasks/`.
+- Pipeline tests: `cognee/tests/unit/modules/pipelines/` (runner semantics,
+  context, provenance, rollback) and `cognee/tests/unit/pipelines/` (the
+  lightweight API).