jinaga-server 3.5.4 → 3.6.0

package/docs/analysis/feed-builder-nested-existential-bug.md

@@ -0,0 +1,191 @@
+# Bug Analysis: Feed Builder Strips Inner Negative Existential Conditions
+
+**Date:** 2026-03-30
+**Area:** `jinaga` package — `feed-builder.js` / `specification-sql.ts`
+**Symptom:** EventName facts not delivered to clients for events that have been deleted and restored
+
+---
+
+## Observed Symptom
+
+The LaunchKings admin portal Events dropdown shows the raw fact hash prefix (e.g. `65bc55e`) instead of the event's human-readable name ("LATAM test"). The `CodeLaunch.Event.Name` fact exists in the database and the distribution rules authorize it, but it is never delivered to the client via feeds.
+
+## Database State That Triggers the Bug
+
+```
+CodeLaunch.Event         1 fact   — id: 65bc55e9-495a-4418-9744-3dea6f8c8f2a
+CodeLaunch.Event.Name    1 fact   — value: "LATAM test"  (EXISTS — not missing)
+CodeLaunch.Event.Delete  2 facts  — deleted twice
+CodeLaunch.Event.Restore 2 facts  — restored twice → event is currently active
+```
+
+The event is logically active (every delete has a matching restore), but its presence of any `EventDelete` facts triggers the bug.
+
+---
+
+## Root Cause
+
+### The Specification
+
+The `eventsListInTenant` specification uses `Event.in(tenant)`:
+
+```typescript
+static in(tenant: LabelOf<Tenant>) {
+    return tenant.successors(Event, event => event.tenant)
+        .notExists(event =>
+            event.successors(EventDelete, d => d.event)
+                .notExists(d =>
+                    d.successors(EventRestore, r => r.eventDelete)
+                )
+        );
+}
+```
+
+The full `!E` condition is:
+```
+!E { EventDelete [ u2->event = u1,  !E { EventRestore [ u3->eventDelete = u2 ] } ] }
+```
+Meaning: *no un-restored EventDelete exists for this event*.
+
+### What `buildFeeds` Produces
+
+`buildFeeds` calls `buildExistentialCondition` to create a simplified version of negative existential conditions for use in feed specifications. The relevant code:
+
+```js
+// feed-builder.js — buildExistentialCondition()
+for (const match of matches) {
+    existentialCondition = {
+        ...existentialCondition,
+        matches: [...existentialCondition.matches, {
+            ...match,
+            conditions: match.conditions.filter(isPathCondition)  // strips ALL existential conditions
+        }]
+    };
+
+    for (const innerExistentialCondition of match.conditions.filter(isExistentialCondition)) {
+        if (innerExistentialCondition.exists) {
+            // ✓ positive inner existentials are recursively included
+        }
+        // ✗ negative inner existentials (exists=false) are silently dropped
+    }
+}
+```
+
+The inner `!E{EventRestore}` condition has `exists=false` and is therefore **dropped**. The simplified condition produced is:
+
+```
+!E { EventDelete [ u2->event = u1 ] }
+```
+Meaning: *no EventDelete at all*.
+
+### The Five Feeds for `eventsListInTenant`
+
+`buildFeeds` produces these feeds:
+
+| Feed | Purpose | Condition on Event |
+|---|---|---|
+| FeedA | Detect EventRestore arrivals | none (bare join) |
+| FeedB | Detect EventDelete arrivals | none (bare join) |
+| FeedC | Deliver Event facts | `!E{EventDelete}` ← simplified |
+| FeedD | Detect EventName supersession | `!E{EventDelete}` ← inherited |
+| FeedE | **Deliver EventName facts** | `!E{EventDelete}` ← **BUG** |
+
+FeedE inherits FeedC's simplified condition via `addProjections(FeedC, ...)`. FeedE's SQL is therefore:
+
+```sql
+SELECT f_name.hash, sort(array[f_name.fact_id], 'desc') as bookmark
+FROM fact f_tenant
+JOIN edge e1 ON e1.predecessor_fact_id = f_tenant.fact_id AND e1.role_id = $event_tenant_role
+JOIN fact f_event ON f_event.fact_id = e1.successor_fact_id
+JOIN edge e2 ON e2.predecessor_fact_id = f_event.fact_id AND e2.role_id = $eventname_event_role
+JOIN fact f_name ON f_name.fact_id = e2.successor_fact_id
+WHERE f_tenant.fact_type_id = $tenant_type AND f_tenant.hash = $tenant_hash
+AND NOT EXISTS (
+    SELECT 1 FROM edge e_del
+    WHERE e_del.predecessor_fact_id = f_event.fact_id
+    AND e_del.role_id = $eventdelete_event_role
+    -- ↑ checks for ANY EventDelete, not just un-restored ones
+)
+AND NOT EXISTS (
+    SELECT 1 FROM edge e_prior
+    WHERE e_prior.predecessor_fact_id = f_name.fact_id
+    AND e_prior.role_id = $eventname_prior_role
+)
+AND sort(array[f_name.fact_id], 'desc') > $bookmark
+ORDER BY bookmark ASC LIMIT 100
+```
+
+Because the "LATAM test" event has `EventDelete` facts (regardless of `EventRestore`), the `NOT EXISTS (EventDelete)` clause is false, the event is excluded, and **the query returns zero rows**.
+
+### Why the Listener Doesn't Rescue It
+
+The observer also registers inverse specification listeners. When FeedA delivers `EventRestore` facts, the listener fires and calls `notifyAdded` — which reads EventName from the **local IndexedDB store**. Because FeedE never delivered EventName to the local store, the re-evaluation finds nothing, and names remain absent.
+
+---
+
+## Impact
+
+Any event that has ever been deleted (even if subsequently restored) will have its name and other projection facts permanently withheld from all projection feeds, while the base `Event` fact itself still appears (for example via FeedC). This affects:
+
+- `CodeLaunch.Event.Name` — event name not shown in nav dropdown
+- `CodeLaunch.Event.Date` — event dates not shown
+- Any other projected facts on events from `eventsListInTenant` or similar specs
+
+In this environment the event was deleted and restored twice, making it the only event and confirming 100% reproduction.
+
+---
+
+## The Fix
+
+### Option A — Support Nested NOT EXISTS in Feed SQL (Correct but Complex)
+
+Extend `buildExistentialCondition` to recursively include negative inner existential conditions:
+
+```js
+// feed-builder.js — buildExistentialCondition()
+for (const innerExistentialCondition of match.conditions.filter(isExistentialCondition)) {
+    if (innerExistentialCondition.exists) {
+        // existing logic — keep positive existentials
+        const { existentialCondition: newEC, ... } = buildExistentialCondition(...);
+        existentialCondition = newEC;
+    } else {
+        // NEW: also recurse into negative existentials
+        const { existentialCondition: newEC, ... } = buildExistentialCondition(
+            innerExistentialCondition, innerExistentialCondition.matches, ...
+        );
+        existentialCondition = { ...existentialCondition, innerConditions: [..., newEC] };
+    }
+}
+```
+
+This also requires:
+1. Adding a `innerConditions` (or `notExistsConditions`) field to `ExistentialConditionDescription`
+2. Updating `generateNotExistsWhereClause` in `specification-sql.ts` to recurse into nested conditions
+3. Updating `notExistsConditionsEqual` in `distribution-engine.ts` if nested conditions need to be compared
+
+### Option B — Omit Outer Not-Exists from Projection Feeds (Simpler)
+
+When `addProjections` builds projection feeds (FeedD, FeedE), start from a version of the specification that **drops all not-exists conditions on the base matches**. Projection feeds would then return facts for all events (active or not), and the full specification read handles the correct filtering.
+
+```js
+// feed-builder.js — buildFeeds()
+const baseFeed = stripNotExistsConditions(finalFeed); // new helper
+const feedsWithProjections = addProjections(baseFeed, unusedGivens, specification.projection.components);
+```
+
+This is safe because:
+- FeedA and FeedB already handle re-evaluation when delete/restore facts arrive
+- The actual filtering is done by `factManager.read()` using the full specification
+- Feeds are designed to be a **superset** of relevant facts; over-delivery is intentional
+
+Option B is simpler, has no SQL changes, and aligns with the documented design principle that "feeds are not executed with nested existential conditions."
+
+---
+
+## Files Involved
+
+| File | Role |
+|---|---|
+| `jinaga/src/specification/feed-builder.ts` | `buildExistentialCondition` — drops inner `!E` conditions |
+| `src/postgres/specification-sql.ts` | `generateNotExistsWhereClause` — only handles flat conditions |
+| `jinaga/src/distribution/distribution-engine.ts` | `notExistsConditionsEqual` — explicitly skips nested comparison |

package/integration-test/jinaga-test/distribution-multi-given.test.js

@@ -0,0 +1,184 @@
+const request = require('supertest');
+const { buildModel, User } = require('jinaga');
+const { createSubscriptionApp, asUser, randomSuffix } = require('./subscription-helpers');
+
+// Exercises a distribution rule whose share/with specs have multiple
+// `given` labels — the "multi-given distribution rule (#161)" dimension
+// the PR claims is closed by canAuthorizeByComposition.
+//
+// The model intentionally pulls BOTH givens into every match clause
+// (Task has predecessors on Tenant *and* Project, ProjectAccess has
+// predecessors on Tenant, Project, and User). This avoids a separate
+// PostgresStore SQL-builder bug that surfaces when a given is referenced
+// only in a WHERE clause with no JOIN — that issue is unrelated to the
+// distribution-engine work in PR #163 and is worth a dedicated bug
+// report rather than masking here.
+
+class Tenant {
+  static Type = "MultiGiven.Tenant";
+  type = Tenant.Type;
+  constructor(creator, identifier) {
+    this.creator = creator;
+    this.identifier = identifier;
+  }
+}
+
+class Project {
+  static Type = "MultiGiven.Project";
+  type = Project.Type;
+  constructor(tenant, identifier) {
+    this.tenant = tenant;
+    this.identifier = identifier;
+  }
+}
+
+class Task {
+  static Type = "MultiGiven.Task";
+  type = Task.Type;
+  constructor(tenant, project, description) {
+    this.tenant = tenant;
+    this.project = project;
+    this.description = description;
+  }
+}
+
+class ProjectAccess {
+  static Type = "MultiGiven.ProjectAccess";
+  type = ProjectAccess.Type;
+  constructor(tenant, project, user) {
+    this.tenant = tenant;
+    this.project = project;
+    this.user = user;
+  }
+}
+
+const model = buildModel(b => b
+  .type(User)
+  .type(Tenant, m => m.predecessor("creator", User))
+  .type(Project, m => m.predecessor("tenant", Tenant))
+  .type(Task, m => m
+    .predecessor("tenant", Tenant)
+    .predecessor("project", Project))
+  .type(ProjectAccess, m => m
+    .predecessor("tenant", Tenant)
+    .predecessor("project", Project)
+    .predecessor("user", User))
+);
+
+function authorization(a) {
+  return a
+    .any(User)
+    .any(Tenant)
+    .any(Project)
+    .any(Task)
+    .any(ProjectAccess);
+}
+
+// Both share and with use a 2-given (Tenant, Project) and reference
+// both labels in their match clauses.
+function distribution(d) {
+  return d
+    .share(model.given(Tenant, Project).match((tenant, project, facts) =>
+      facts.ofType(Task)
+        .join(t => t.tenant, tenant)
+        .join(t => t.project, project)
+    ))
+    .with(model.given(Tenant, Project).match((tenant, project, facts) =>
+      facts.ofType(ProjectAccess)
+        .join(a => a.tenant, tenant)
+        .join(a => a.project, project)
+        .selectMany(a => facts.ofType(User).join(u => u, a.user))
+    ));
+}
+
+function taskSpecText(tenantHash, projectHash) {
+  return `let p1: ${Tenant.Type} = #${tenantHash}\n` +
+    `let p2: ${Project.Type} = #${projectHash}\n` +
+    `(p1: ${Tenant.Type}, p2: ${Project.Type}) {\n` +
+    `    t: ${Task.Type} [\n` +
+    `        t->tenant: ${Tenant.Type} = p1\n` +
+    `        t->project: ${Project.Type} = p2\n` +
+    `    ]\n` +
+    `} => t`;
+}
+
+describe('Distribution rule with multi-given share/with', () => {
+  let app, withSession, close;
+  let tenantValue, projectValue, tenantHash, projectHash, taskHash;
+  let creatorId;
+
+  beforeEach(async () => {
+    ({ app, withSession, close } = await createSubscriptionApp({
+      model, authorization, distribution
+    }));
+
+    creatorId = 'mg-creator-' + randomSuffix();
+    ({ tenantHash, projectHash, taskHash } = await asUser(withSession, creatorId, async (j) => {
+      const { userFact: creatorFact } = await j.login();
+      const tenant = await j.fact(new Tenant(creatorFact, 't-' + randomSuffix()));
+      tenantValue = tenant;
+      const project = await j.fact(new Project(tenant, 'p-' + randomSuffix()));
+      projectValue = project;
+      const task = await j.fact(new Task(tenant, project, 'do the thing'));
+      return {
+        tenantHash: j.hash(tenant),
+        projectHash: j.hash(project),
+        taskHash: j.hash(task)
+      };
+    }));
+  });
+
+  afterEach(async () => {
+    if (close) await close();
+  });
+
+  it('delivers tasks to a user with ProjectAccess through a 2-given subscription', async () => {
+    const memberId = 'mg-member-' + randomSuffix();
+    const memberFact = await asUser(withSession, memberId, async (j) => {
+      const { userFact } = await j.login();
+      return userFact;
+    });
+
+    // Grant ProjectAccess that covers both the tenant and the project.
+    await asUser(withSession, creatorId, async (j) => {
+      await j.fact(new ProjectAccess(tenantValue, projectValue, memberFact));
+    });
+
+    const feedsResponse = await request(app)
+      .post('/feeds')
+      .set('Content-Type', 'text/plain')
+      .set('x-test-user-id', memberId)
+      .send(taskSpecText(tenantHash, projectHash));
+    expect(feedsResponse.status).toBe(200);
+    const { feeds } = JSON.parse(feedsResponse.text);
+    expect(feeds.length).toBeGreaterThan(0);
+
+    const pollResponse = await request(app)
+      .get(`/feeds/${feeds[0]}`)
+      .set('x-test-user-id', memberId);
+    expect(pollResponse.status).toBe(200);
+    const body = JSON.parse(pollResponse.text);
+    const taskRefs = (body.references || []).filter(r => r.type === Task.Type);
+    expect(taskRefs.map(r => r.hash)).toContain(taskHash);
+  });
+
+  it('hides tasks from a user without ProjectAccess through the same 2-given subscription', async () => {
+    const outsiderId = 'mg-outsider-' + randomSuffix();
+    await asUser(withSession, outsiderId, async (j) => { /* login only */ });
+
+    const feedsResponse = await request(app)
+      .post('/feeds')
+      .set('Content-Type', 'text/plain')
+      .set('x-test-user-id', outsiderId)
+      .send(taskSpecText(tenantHash, projectHash));
+    expect(feedsResponse.status).toBe(200);
+    const { feeds } = JSON.parse(feedsResponse.text);
+
+    const pollResponse = await request(app)
+      .get(`/feeds/${feeds[0]}`)
+      .set('x-test-user-id', outsiderId);
+    expect(pollResponse.status).toBe(200);
+    const body = JSON.parse(pollResponse.text);
+    expect((body.references || []).filter(r => r.type === Task.Type)).toHaveLength(0);
+  });
+});

package/integration-test/jinaga-test/distribution-negating-feed.test.js

@@ -0,0 +1,202 @@
+const request = require('supertest');
+const { buildModel, User } = require('jinaga');
+const { createSubscriptionApp, asUser, randomSuffix } = require('./subscription-helpers');
+
+// Exercises a distribution rule whose user-spec uses notExists — the
+// "negating-feed authorization (#196)" dimension the PR closes by
+// having DistributionEngine.canDistributeTo fall back to
+// canAuthorizeByComposition. We avoid the late-auth/intersection path
+// here so the test is hitting canDistributeTo (via feedWithDistribution)
+// directly with a negating user-spec.
+
+jest.setTimeout(30000);
+
+class Group {
+  static Type = "NegatingFeed.Group";
+  type = Group.Type;
+  constructor(creator, name) {
+    this.creator = creator;
+    this.name = name;
+  }
+}
+
+class Member {
+  static Type = "NegatingFeed.Member";
+  type = Member.Type;
+  constructor(group, user) {
+    this.group = group;
+    this.user = user;
+  }
+}
+
+class MemberRemoved {
+  static Type = "NegatingFeed.Member.Removed";
+  type = MemberRemoved.Type;
+  constructor(member) {
+    this.member = member;
+  }
+}
+
+class Content {
+  static Type = "NegatingFeed.Content";
+  type = Content.Type;
+  constructor(group, body) {
+    this.group = group;
+    this.body = body;
+  }
+}
+
+const model = buildModel(b => b
+  .type(User)
+  .type(Group, m => m.predecessor("creator", User))
+  .type(Member, m => m
+    .predecessor("group", Group)
+    .predecessor("user", User))
+  .type(MemberRemoved, m => m.predecessor("member", Member))
+  .type(Content, m => m.predecessor("group", Group))
+);
+
+function authorization(a) {
+  return a
+    .any(User)
+    .any(Group)
+    .any(Member)
+    .any(MemberRemoved)
+    .any(Content);
+}
+
+// Share Content of a Group with users who are currently Members — i.e.
+// have a Member fact with no MemberRemoved successor. The notExists
+// here is what makes this a "negating-feed" user-spec.
+function distribution(d) {
+  return d
+    .share(model.given(Group).match((g, facts) =>
+      facts.ofType(Content).join(c => c.group, g)
+    ))
+    .with(model.given(Group).match((g, facts) =>
+      facts.ofType(Member)
+        .join(m => m.group, g)
+        .notExists(m => facts.ofType(MemberRemoved).join(r => r.member, m))
+        .selectMany(m => facts.ofType(User).join(u => u, m.user))
+    ));
+}
+
+function contentSpecText(groupHash) {
+  return `let p1: ${Group.Type} = #${groupHash}\n` +
+    `(p1: ${Group.Type}) {\n` +
+    `    c: ${Content.Type} [\n` +
+    `        c->group: ${Group.Type} = p1\n` +
+    `    ]\n` +
+    `} => c`;
+}
+
+describe('Distribution rule with negating feed (notExists)', () => {
+  let app, withSession, close;
+  let groupValue, groupHash, contentHash;
+  let creatorId;
+
+  beforeEach(async () => {
+    ({ app, withSession, close } = await createSubscriptionApp({
+      model, authorization, distribution
+    }));
+
+    creatorId = 'group-creator-' + randomSuffix();
+    ({ groupHash, contentHash } = await asUser(withSession, creatorId, async (creatorJ) => {
+      const { userFact: creatorFact } = await creatorJ.login();
+      const group = await creatorJ.fact(new Group(creatorFact, 'g-' + randomSuffix()));
+      groupValue = group;
+      const content = await creatorJ.fact(new Content(group, 'shared body'));
+      return {
+        groupHash: creatorJ.hash(group),
+        contentHash: creatorJ.hash(content)
+      };
+    }));
+  });
+
+  afterEach(async () => {
+    if (close) await close();
+  });
+
+  it('delivers content to a current member', async () => {
+    const memberId = 'group-member-' + randomSuffix();
+    const memberFact = await asUser(withSession, memberId, async (j) => {
+      const { userFact } = await j.login();
+      return userFact;
+    });
+
+    // Grant membership BEFORE subscribing so the distribution check
+    // (notExists removed → current member) passes and we never hit
+    // intersection. This makes the test target canDistributeTo
+    // directly via feedWithDistribution.
+    await asUser(withSession, creatorId, async (j) => {
+      await j.fact(new Member(groupValue, memberFact));
+    });
+
+    const feedsResponse = await request(app)
+      .post('/feeds')
+      .set('Content-Type', 'text/plain')
+      .set('x-test-user-id', memberId)
+      .send(contentSpecText(groupHash));
+    expect(feedsResponse.status).toBe(200);
+    const { feeds } = JSON.parse(feedsResponse.text);
+
+    const pollResponse = await request(app)
+      .get(`/feeds/${feeds[0]}`)
+      .set('x-test-user-id', memberId);
+    expect(pollResponse.status).toBe(200);
+    const body = JSON.parse(pollResponse.text);
+    const visible = (body.references || []).filter(r => r.type === Content.Type);
+    expect(visible.map(r => r.hash)).toContain(contentHash);
+  });
+
+  it('hides content from a removed member', async () => {
+    const memberId = 'group-removed-' + randomSuffix();
+    const memberFact = await asUser(withSession, memberId, async (j) => {
+      const { userFact } = await j.login();
+      return userFact;
+    });
+
+    // Add then remove the membership. The notExists branch flips,
+    // canDistributeTo denies, and the poll returns an empty page.
+    let memberValue;
+    await asUser(withSession, creatorId, async (j) => {
+      memberValue = await j.fact(new Member(groupValue, memberFact));
+      await j.fact(new MemberRemoved(memberValue));
+    });
+
+    const feedsResponse = await request(app)
+      .post('/feeds')
+      .set('Content-Type', 'text/plain')
+      .set('x-test-user-id', memberId)
+      .send(contentSpecText(groupHash));
+    expect(feedsResponse.status).toBe(200);
+    const { feeds } = JSON.parse(feedsResponse.text);
+
+    const pollResponse = await request(app)
+      .get(`/feeds/${feeds[0]}`)
+      .set('x-test-user-id', memberId);
+    expect(pollResponse.status).toBe(200);
+    const body = JSON.parse(pollResponse.text);
+    expect((body.references || []).filter(r => r.type === Content.Type)).toHaveLength(0);
+  });
+
+  it('hides content from a never-member', async () => {
+    const outsiderId = 'group-outsider-' + randomSuffix();
+    await asUser(withSession, outsiderId, async (j) => { /* login only */ });
+
+    const feedsResponse = await request(app)
+      .post('/feeds')
+      .set('Content-Type', 'text/plain')
+      .set('x-test-user-id', outsiderId)
+      .send(contentSpecText(groupHash));
+    expect(feedsResponse.status).toBe(200);
+    const { feeds } = JSON.parse(feedsResponse.text);
+
+    const pollResponse = await request(app)
+      .get(`/feeds/${feeds[0]}`)
+      .set('x-test-user-id', outsiderId);
+    expect(pollResponse.status).toBe(200);
+    const body = JSON.parse(pollResponse.text);
+    expect((body.references || []).filter(r => r.type === Content.Type)).toHaveLength(0);
+  });
+});