@pulseindex/sdk 4.0.0 → 5.0.0

package/CHANGELOG.md

@@ -1,5 +1,120 @@
 # Changelog
 
+## 5.0.0
+
+Needs a PulseIndex engine at v2.0.0 or later. The wire contract is a
+breaking change: an SDK at 4.x cannot talk to a v2 engine, and this cannot
+talk to a v1 one.
+
+### A radius that means what it says, and "the nearest K"
+
+A circle used to be a union of geohash cells, and a union of cells is a superset
+of the circle. Measured against a million entities, a 1 km search returned
+**2,479** rows where **1,241** were really inside — and nothing in the answer
+said which. You hydrated all 2,479 from your own store and measured them again.
+
+Send a position with the record and the engine settles the edge itself:
+
+    points: { where: { lat: 41.0369, lon: 28.9850 } }
+
+Then `withinRadius({ ..., field: 'where' })` narrows on the cells and measures
+the true distance, or `within('where', lat, lon, km)` measures without them.
+Degrees go on the wire and the engine packs them: a representation split between
+this package and the engine, with nothing comparing the two, is how the geo
+defects in 4.0.0 happened.
+
+`nearest('where', lat, lon)` orders by distance, nearest first. That was not a
+question you could ask before at any price. It needs no radius guessed to make
+it quick, and combines with every other filter, so "the closest ten available
+drivers with room" is one request.
+
+Ordering is to the centimetre, which is the precision a stored position has.
+
+
+### The result count now says whether it is the whole count
+
+A paged search stops as soon as the page is full, so the `totalMatches` it
+carried was only what the engine had reached by then. Nothing said so, and it
+does not look partial: measured on 500,000 records that all matched, a page of
+20 reported **65,536**; a harder query reported **48,105** against a true total
+of **333,895**. Any interface printing "N results" from that was wrong by
+several times over and looked fine.
+
+`totalIsExact` now comes from the engine instead of being guessed from
+`limit === 0`, and `searchWithTotal()` is **one request instead of two** — the
+wire can ask for a page and a true count together. A page whose matches all fit
+inside it is reported exact, which the old guess got wrong.
+
+
+### A record carries whatever numbers you name, and no field the engine chose
+
+An entity used to be forced through one `uint32` called `price` and one `uint64`
+bitfield called `locationPrefix`. That is a schema this SDK had no business
+imposing: one number per record, under a name we picked, with no negatives, and
+nothing above 4,294,967,295. A rating, a capacity, a timestamp, an elevation and
+a balance were all the same field or no field at all.
+
+Both are gone. A record now carries `numbers`, under your own names:
+
+    client.index('1001', {
+      categories: ['feature:pool'],
+      numbers: { price_cents: 45000, bedrooms: 3, built_at: 1712000000 },
+    })
+
+Any name, any 64-bit integer, any number of them, and every one is filterable
+through `range()` and orderable through `sortBy()`. A name means nothing to the
+engine beyond its hash.
+
+### Three things this fixes that were losing data quietly
+
+**A fraction was floored without a word.** `4.3` was sent as `4`, `0.5` as `0`,
+`199.99` as `199` — measured, not inferred. The engine's column is a 64-bit
+integer, so a fraction is refused now, with the scaling it needs named. Keep the
+scale on your side: a price in cents, a rating out of 100.
+
+**A field of your own called `price` disappeared.** Sixteen key names were
+reserved out of your attributes, and anything under one was dropped: measured,
+`{price: 250, rating: 4.3, lat: 41, kind: 'villa'}` came out as
+`["rating:4.3", "kind:villa"]` — `price` and `lat` gone, no error. Numbers go
+through `numbers` now, so nothing in your own object is swallowed.
+
+**A range or an order on a field nothing carries was answered, not refused.**
+It excluded every entity, or left the page in insertion order and reported it as
+sorted. At ten million records `bedrooms 3..6` returned 0 while the tag
+`bedrooms:3` returned 1,666,667. The engine refuses it by name now — but only
+when the tenant holds entities and none of them carries that field, because an
+empty tenant has nothing to be wrong about.
+
+### Migrating
+
+- `index(id, {price: N})` → `index(id, {numbers: {price: N}})`. A top-level
+  `price` is no longer special; it becomes the tag `price:N` like any other
+  scalar attribute.
+- `indexEntity(id, categories, price, locationPrefix, tenantId)` →
+  `indexEntity(id, categories, numbers, tenantId)`.
+- `.location(prefix)` on the query builder is gone. Nothing ever sent it: both
+  SDKs passed 0 on every request.
+- A range bound may now be negative or past 4,294,967,295.
+
+This needs an engine built from the same commit. The proto is a breaking change:
+field numbers 2 and 3 on `IndexEntityRequest`, and 1 on `SearchQueryRequest`, are
+reserved rather than reused.
+
+## 4.0.1
+
+### The covering threshold, corrected against a real app
+
+`withinRadius` is a pre-filter the caller narrows exactly afterwards, so excess
+area is cheaper than predicates. 4.0.0's threshold was too strict: it rejected
+the coarse cell at 5 km, turning a 10-cell covering into 167. Measured through
+the sibling PHP SDK against a real application with 100,000 properties, a 5 km
+radius went from 3,082 µs to 926 µs on wall time — the cost is the request, not
+the search.
+
+The coarse cell is now taken up to 3.0× the circle, which still rejects it at
+2 km where it wastes 4.73× to 6.91×. Both SDKs are regenerated against one
+shared vector fixture, so they still agree cell for cell.
+
 ## 4.0.0
 
 **A major, not a minor.** The previous draft of these notes said 3.2.0. Checking

package/dist/index.d.mts

@@ -27,19 +27,38 @@ interface SortSpec {
     field: string;
     /** Largest first when true; smallest first otherwise. */
     descending: boolean;
+    /**
+     * Order by distance from the query's geo predicate instead of by a field
+     * value. `field` is ignored.
+     *
+     * Ordering is to the centimetre, which is the precision a stored position
+     * has. Rows closer together than that tie, and a tie breaks on the entity id
+     * so the same query returns the same page.
+     */
+    byDistance?: boolean;
 }
 interface RangePredicate {
     field: string;
     minVal: number;
     maxVal: number;
 }
+/** A circle, and the position field to measure it against. */
+interface GeoPredicate {
+    field: string;
+    lat: number;
+    lon: number;
+    /** Inclusive. 0 means no radius bound, only an origin to measure from. */
+    radiusKm: number;
+}
 interface SearchQueryRequest {
-    locationPrefix: string;
     filters: FilterPredicate[];
+    geo?: GeoPredicate;
     ranges: RangePredicate[];
     limit: number;
     offset: number;
     tenantId: string;
+    /** Count every match rather than stopping when the page is full. */
+    exactTotal: boolean;
     /** Absent returns matches in entity-id order. */
     sort?: SortSpec;
 }
@@ -61,8 +80,8 @@ interface SearchResponse {
 }
 interface IndexEntityRequest {
     entityId: string;
-    locationPrefix: string;
-    price: number;
+    numbers: Record<string, number>;
+    points: Record<string, GeoPoint>;
     categories: string[];
     tenantId: string;
 }
@@ -88,10 +107,21 @@ interface RadiusOptions {
     lon?: number;
     radiusKm: number;
     precision?: number;
+    /**
+     * The position field to measure against, as named in `points` when indexing.
+     *
+     * Given one, the engine narrows on the geohash cells and then measures the
+     * true distance, so the answer holds only what is really inside the circle.
+     * Without it the cells are the whole answer, and a union of cells is a
+     * superset: measured at a million entities, a 1 km search returned 2,479
+     * rows where 1,241 were inside.
+     */
+    field?: string;
 }
 interface SearchRequestOptions {
     tenantId?: string;
-    locationPrefix?: EntityId;
+    /** Count every match rather than stopping when the page is full. */
+    exactTotal?: boolean;
     must?: string | string[];
     should?: string | string[];
     mustNot?: string | string[];
@@ -118,9 +148,7 @@ interface SearchRequestOptions {
 interface EntityAttributes {
     categories?: unknown;
     tags?: unknown;
-    price?: unknown;
-    locationPrefix?: unknown;
-    location_prefix?: unknown;
+    numbers?: unknown;
     tenantId?: unknown;
     tenant_id?: unknown;
     latitude?: unknown;
@@ -137,9 +165,7 @@ interface EntityInput {
     attributes?: EntityAttributes;
     categories?: unknown;
     tags?: unknown;
-    price?: unknown;
-    locationPrefix?: unknown;
-    location_prefix?: unknown;
+    numbers?: unknown;
     tenantId?: unknown;
     tenant_id?: unknown;
     latitude?: unknown;
@@ -156,11 +182,21 @@ interface BatchEntityInput {
     attributes?: EntityAttributes;
     [key: string]: unknown;
 }
+/** One position in degrees. The engine packs it; this SDK does not. */
+interface GeoPoint {
+    lat: number;
+    lon: number;
+}
 interface EncodedEntity {
     entityId: string;
     categories: string[];
-    price: number;
-    locationPrefix: string;
+    /** Positions under your own names. */
+    points: Record<string, GeoPoint>;
+    /**
+     * Numeric fields under your own names. Any name, any integer, any number of
+     * them. This replaced a single `price` field the engine named for you.
+     */
+    numbers: Record<string, number>;
     tenantId: string;
 }
 interface PulseIndexClientConfig {
@@ -198,7 +234,15 @@ declare class QueryBuilder {
     private state;
     constructor(executor?: QueryExecutor | null);
     tenant(tenantId: string): QueryBuilder;
-    location(locationPrefix: string | number | bigint): QueryBuilder;
+    /**
+     * Count every match instead of stopping as soon as the page is full.
+     *
+     * A paged search stops early, so the `totalMatches` it carries is only what
+     * the engine had counted by then — a lower bound, and one that does not look
+     * like one. This makes the count exact in the same request; `totalIsExact`
+     * on the response says which you got.
+     */
+    exactTotal(enabled?: boolean): QueryBuilder;
     must(attribute: string | string[]): QueryBuilder;
     /**
      * At least one of these has to match.
@@ -216,6 +260,15 @@ declare class QueryBuilder {
     inGeoHash(geohash: string): QueryBuilder;
     withinRadius(lat: number, lon: number, radiusKm: number, precision?: number): QueryBuilder;
     withinRadius(options: RadiusOptions): QueryBuilder;
+    /**
+     * Filter on a numeric field's inclusive range.
+     *
+     * `price` is the only number an entity carries. Naming any other field is
+     * refused by the engine rather than answered, because a field nothing carries
+     * can only match nothing, and an empty page looks exactly like a real one.
+     * Model any other number as a category token instead: `must('bedrooms:3')`,
+     * or several in one SHOULD group for a range of values.
+     */
     range(field: string, min: number, max: number): QueryBuilder;
     /**
      * How many ids to return. Zero asks the engine for the number of matches
@@ -234,11 +287,31 @@ declare class QueryBuilder {
     /** Order the page by a numeric field, largest first. */
     sortDesc(field: string): QueryBuilder;
     /**
-     * Order the page by a numeric field. Rows carrying no value for it sort last
-     * in both directions; they still count towards `totalMatches`, they simply
-     * have nothing to be ordered by.
+     * Order the page by a numeric field.
+     *
+     * Bounded exactly as {@link range} is: `price` is the only field an entity
+     * carries, and any other name is refused rather than silently ignored. An
+     * order by a field nothing carries used to leave the page in insertion order
+     * and report it as sorted.
      */
     sortBy(field: string, descending?: boolean): QueryBuilder;
+    /**
+     * Keep only entities within `radiusKm` of the point, measured exactly.
+     *
+     * This is the circle on its own. {@link withinRadius} with a `field` adds
+     * the geohash cells too, which is what stops the engine opening every part
+     * of the index to find them.
+     */
+    within(field: string, lat: number, lon: number, radiusKm: number): QueryBuilder;
+    /**
+     * Order the page by distance from the point, nearest first.
+     *
+     * Without a radius this is "the nearest K of whatever else matched"; combine
+     * it with {@link within} or {@link withinRadius} to bound the search as well.
+     * It used to be impossible: a radius returned everything inside it unordered,
+     * so you hydrated every id from your own store before you could sort them.
+     */
+    nearest(field: string, lat: number, lon: number): QueryBuilder;
     toRequest(defaultTenantId?: string): SearchQueryRequest;
     toArray(defaultTenantId?: string): SearchQueryRequest;
     execute(): Promise<SearchResponse>;
@@ -318,12 +391,21 @@ declare class PulseIndexClient implements QueryExecutor {
      * reported 10,866 for a page of 100. Anything that prints "page 1 of N" from
      * that number is wrong by an order of magnitude and looks fine.
      *
-     * This sends the count query as well, so it costs two round trips and returns
-     * a total you can divide by a page size.
+     * One request. This used to run the whole query twice — once for the page,
+     * once for the count — because the wire had no way to ask for both. It does
+     * now, so this is the same round trip with `exactTotal` set, and the total
+     * you get back can be divided by a page size.
      */
     searchWithTotal(query: QueryBuilder | SearchRequestOptions): Promise<SearchResponse>;
     index(entityIdOrInput: EntityId | EntityInput, attributes?: EntityAttributes): Promise<IndexEntityResponse>;
-    indexEntity(entityId: EntityId, categories?: string[], price?: number, locationPrefix?: EntityId, tenantId?: string): Promise<boolean>;
+    /**
+     * Index one record.
+     *
+     * `numbers` are yours to name: `{price_cents: 45000, bedrooms: 3}`. This
+     * used to take a single `price` and a `locationPrefix`, which was a schema
+     * this SDK had no business imposing.
+     */
+    indexEntity(entityId: EntityId, categories?: string[], numbers?: Record<string, number>, tenantId?: string): Promise<boolean>;
     batchIndex(entities: Array<EntityInput | BatchEntityInput>): Promise<BatchIndexResponse>;
     delete(entityId: EntityId, tenantId?: string): Promise<DeleteResponse>;
     deleteEntity(entityId: EntityId, tenantId?: string): Promise<boolean>;
@@ -402,11 +484,17 @@ declare class GeoHash {
      * How much area outside the circle a covering may carry before a finer
      * precision is worth its cell count.
      *
-     * 2.0 is where the measured choices come out right at every radius: it
-     * rejects the coarse cell at 2 km (6.91x) and 5 km (2.76x) and accepts it at
-     * 15 km (1.44x), which is also where the cell count turns from 47 into 1,120.
+     * `withinRadius` is a pre-filter the caller narrows exactly afterwards, so
+     * excess area is cheaper than predicates.
+     *
+     * 2.0 was the first attempt and it was too strict. It rejected the coarse
+     * cell at 5 km, so a covering that had cost 10 cells cost 167, and the demo
+     * benchmark went from beating PostgreSQL to losing to it by 2.76x on wall
+     * time — the cost is the request, not the search. 3.0 keeps the cheap
+     * covering at 5 km (2.76x) and still rejects the coarse cell at 2 km, where
+     * it wastes 4.73x to 6.91x depending on latitude.
      */
-    static readonly ACCEPTABLE_COVER_RATIO = 2;
+    static readonly ACCEPTABLE_COVER_RATIO = 3;
     private static readonly NEIGHBORS;
     private static readonly BORDERS;
     static encode(lat: number, lon: number, precision?: number): string;

package/dist/index.d.ts

@@ -27,19 +27,38 @@ interface SortSpec {
     field: string;
     /** Largest first when true; smallest first otherwise. */
     descending: boolean;
+    /**
+     * Order by distance from the query's geo predicate instead of by a field
+     * value. `field` is ignored.
+     *
+     * Ordering is to the centimetre, which is the precision a stored position
+     * has. Rows closer together than that tie, and a tie breaks on the entity id
+     * so the same query returns the same page.
+     */
+    byDistance?: boolean;
 }
 interface RangePredicate {
     field: string;
     minVal: number;
     maxVal: number;
 }
+/** A circle, and the position field to measure it against. */
+interface GeoPredicate {
+    field: string;
+    lat: number;
+    lon: number;
+    /** Inclusive. 0 means no radius bound, only an origin to measure from. */
+    radiusKm: number;
+}
 interface SearchQueryRequest {
-    locationPrefix: string;
     filters: FilterPredicate[];
+    geo?: GeoPredicate;
     ranges: RangePredicate[];
     limit: number;
     offset: number;
     tenantId: string;
+    /** Count every match rather than stopping when the page is full. */
+    exactTotal: boolean;
     /** Absent returns matches in entity-id order. */
     sort?: SortSpec;
 }
@@ -61,8 +80,8 @@ interface SearchResponse {
 }
 interface IndexEntityRequest {
     entityId: string;
-    locationPrefix: string;
-    price: number;
+    numbers: Record<string, number>;
+    points: Record<string, GeoPoint>;
     categories: string[];
     tenantId: string;
 }
@@ -88,10 +107,21 @@ interface RadiusOptions {
     lon?: number;
     radiusKm: number;
     precision?: number;
+    /**
+     * The position field to measure against, as named in `points` when indexing.
+     *
+     * Given one, the engine narrows on the geohash cells and then measures the
+     * true distance, so the answer holds only what is really inside the circle.
+     * Without it the cells are the whole answer, and a union of cells is a
+     * superset: measured at a million entities, a 1 km search returned 2,479
+     * rows where 1,241 were inside.
+     */
+    field?: string;
 }
 interface SearchRequestOptions {
     tenantId?: string;
-    locationPrefix?: EntityId;
+    /** Count every match rather than stopping when the page is full. */
+    exactTotal?: boolean;
     must?: string | string[];
     should?: string | string[];
     mustNot?: string | string[];
@@ -118,9 +148,7 @@ interface SearchRequestOptions {
 interface EntityAttributes {
     categories?: unknown;
     tags?: unknown;
-    price?: unknown;
-    locationPrefix?: unknown;
-    location_prefix?: unknown;
+    numbers?: unknown;
     tenantId?: unknown;
     tenant_id?: unknown;
     latitude?: unknown;
@@ -137,9 +165,7 @@ interface EntityInput {
     attributes?: EntityAttributes;
     categories?: unknown;
     tags?: unknown;
-    price?: unknown;
-    locationPrefix?: unknown;
-    location_prefix?: unknown;
+    numbers?: unknown;
     tenantId?: unknown;
     tenant_id?: unknown;
     latitude?: unknown;
@@ -156,11 +182,21 @@ interface BatchEntityInput {
     attributes?: EntityAttributes;
     [key: string]: unknown;
 }
+/** One position in degrees. The engine packs it; this SDK does not. */
+interface GeoPoint {
+    lat: number;
+    lon: number;
+}
 interface EncodedEntity {
     entityId: string;
     categories: string[];
-    price: number;
-    locationPrefix: string;
+    /** Positions under your own names. */
+    points: Record<string, GeoPoint>;
+    /**
+     * Numeric fields under your own names. Any name, any integer, any number of
+     * them. This replaced a single `price` field the engine named for you.
+     */
+    numbers: Record<string, number>;
     tenantId: string;
 }
 interface PulseIndexClientConfig {
@@ -198,7 +234,15 @@ declare class QueryBuilder {
     private state;
     constructor(executor?: QueryExecutor | null);
     tenant(tenantId: string): QueryBuilder;
-    location(locationPrefix: string | number | bigint): QueryBuilder;
+    /**
+     * Count every match instead of stopping as soon as the page is full.
+     *
+     * A paged search stops early, so the `totalMatches` it carries is only what
+     * the engine had counted by then — a lower bound, and one that does not look
+     * like one. This makes the count exact in the same request; `totalIsExact`
+     * on the response says which you got.
+     */
+    exactTotal(enabled?: boolean): QueryBuilder;
     must(attribute: string | string[]): QueryBuilder;
     /**
      * At least one of these has to match.
@@ -216,6 +260,15 @@ declare class QueryBuilder {
     inGeoHash(geohash: string): QueryBuilder;
     withinRadius(lat: number, lon: number, radiusKm: number, precision?: number): QueryBuilder;
     withinRadius(options: RadiusOptions): QueryBuilder;
+    /**
+     * Filter on a numeric field's inclusive range.
+     *
+     * `price` is the only number an entity carries. Naming any other field is
+     * refused by the engine rather than answered, because a field nothing carries
+     * can only match nothing, and an empty page looks exactly like a real one.
+     * Model any other number as a category token instead: `must('bedrooms:3')`,
+     * or several in one SHOULD group for a range of values.
+     */
     range(field: string, min: number, max: number): QueryBuilder;
     /**
      * How many ids to return. Zero asks the engine for the number of matches
@@ -234,11 +287,31 @@ declare class QueryBuilder {
     /** Order the page by a numeric field, largest first. */
     sortDesc(field: string): QueryBuilder;
     /**
-     * Order the page by a numeric field. Rows carrying no value for it sort last
-     * in both directions; they still count towards `totalMatches`, they simply
-     * have nothing to be ordered by.
+     * Order the page by a numeric field.
+     *
+     * Bounded exactly as {@link range} is: `price` is the only field an entity
+     * carries, and any other name is refused rather than silently ignored. An
+     * order by a field nothing carries used to leave the page in insertion order
+     * and report it as sorted.
      */
     sortBy(field: string, descending?: boolean): QueryBuilder;
+    /**
+     * Keep only entities within `radiusKm` of the point, measured exactly.
+     *
+     * This is the circle on its own. {@link withinRadius} with a `field` adds
+     * the geohash cells too, which is what stops the engine opening every part
+     * of the index to find them.
+     */
+    within(field: string, lat: number, lon: number, radiusKm: number): QueryBuilder;
+    /**
+     * Order the page by distance from the point, nearest first.
+     *
+     * Without a radius this is "the nearest K of whatever else matched"; combine
+     * it with {@link within} or {@link withinRadius} to bound the search as well.
+     * It used to be impossible: a radius returned everything inside it unordered,
+     * so you hydrated every id from your own store before you could sort them.
+     */
+    nearest(field: string, lat: number, lon: number): QueryBuilder;
     toRequest(defaultTenantId?: string): SearchQueryRequest;
     toArray(defaultTenantId?: string): SearchQueryRequest;
     execute(): Promise<SearchResponse>;
@@ -318,12 +391,21 @@ declare class PulseIndexClient implements QueryExecutor {
      * reported 10,866 for a page of 100. Anything that prints "page 1 of N" from
      * that number is wrong by an order of magnitude and looks fine.
      *
-     * This sends the count query as well, so it costs two round trips and returns
-     * a total you can divide by a page size.
+     * One request. This used to run the whole query twice — once for the page,
+     * once for the count — because the wire had no way to ask for both. It does
+     * now, so this is the same round trip with `exactTotal` set, and the total
+     * you get back can be divided by a page size.
      */
     searchWithTotal(query: QueryBuilder | SearchRequestOptions): Promise<SearchResponse>;
     index(entityIdOrInput: EntityId | EntityInput, attributes?: EntityAttributes): Promise<IndexEntityResponse>;
-    indexEntity(entityId: EntityId, categories?: string[], price?: number, locationPrefix?: EntityId, tenantId?: string): Promise<boolean>;
+    /**
+     * Index one record.
+     *
+     * `numbers` are yours to name: `{price_cents: 45000, bedrooms: 3}`. This
+     * used to take a single `price` and a `locationPrefix`, which was a schema
+     * this SDK had no business imposing.
+     */
+    indexEntity(entityId: EntityId, categories?: string[], numbers?: Record<string, number>, tenantId?: string): Promise<boolean>;
     batchIndex(entities: Array<EntityInput | BatchEntityInput>): Promise<BatchIndexResponse>;
     delete(entityId: EntityId, tenantId?: string): Promise<DeleteResponse>;
     deleteEntity(entityId: EntityId, tenantId?: string): Promise<boolean>;
@@ -402,11 +484,17 @@ declare class GeoHash {
      * How much area outside the circle a covering may carry before a finer
      * precision is worth its cell count.
      *
-     * 2.0 is where the measured choices come out right at every radius: it
-     * rejects the coarse cell at 2 km (6.91x) and 5 km (2.76x) and accepts it at
-     * 15 km (1.44x), which is also where the cell count turns from 47 into 1,120.
+     * `withinRadius` is a pre-filter the caller narrows exactly afterwards, so
+     * excess area is cheaper than predicates.
+     *
+     * 2.0 was the first attempt and it was too strict. It rejected the coarse
+     * cell at 5 km, so a covering that had cost 10 cells cost 167, and the demo
+     * benchmark went from beating PostgreSQL to losing to it by 2.76x on wall
+     * time — the cost is the request, not the search. 3.0 keeps the cheap
+     * covering at 5 km (2.76x) and still rejects the coarse cell at 2 km, where
+     * it wastes 4.73x to 6.91x depending on latitude.
      */
-    static readonly ACCEPTABLE_COVER_RATIO = 2;
+    static readonly ACCEPTABLE_COVER_RATIO = 3;
     private static readonly NEIGHBORS;
     private static readonly BORDERS;
     static encode(lat: number, lon: number, precision?: number): string;