datahike-browser-tests 1.0.0

package/doc/time_variance.md

@@ -0,0 +1,325 @@
+# Time Variance
+
+As a [temporal database](https://en.wikipedia.org/wiki/Temporal_database), Datahike tracks transaction time for every change, enabling auditing, analytics, and time-travel queries. Each transaction records `:db/txInstant`, allowing you to view data at different points in time.
+
+## Time Travel Views
+
+| View | Function | Returns | Use Case |
+|------|----------|---------|----------|
+| **Current** | `@conn` or `(d/db conn)` | Latest state | Normal queries |
+| **As-of** | `(d/as-of db date)` | State at specific time | "What did the data look like on July 1st?" |
+| **History** | `(d/history db)` | All versions (current + historical) | Audit trails, change analysis |
+| **Since** | `(d/since db date)` | Changes after specific time | "What changed since yesterday?" |
+
+**Related:** For git-like branching and merging of database snapshots, see [Versioning](versioning.md). For removing old historical data from storage, see [Garbage Collection](gc.md).
+
+## Disabling History
+
+If you don't need time-travel queries, disable history tracking to save storage:
+
+```clojure
+(require '[datahike.api :as d])
+
+(d/create-database {:store {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440000"} :keep-history? false})
+```
+
+**Trade-off:** Saves storage and improves write performance, but removes as-of/history/since queries and purging capabilities.
+
+## Setup for Examples
+
+All examples below use this shared setup:
+
+```clojure
+(require '[datahike.api :as d])
+
+;; Simple schema for person data
+(def schema [{:db/ident :name
+              :db/valueType :db.type/string
+              :db/unique :db.unique/identity
+              :db/index true
+              :db/cardinality :db.cardinality/one}
+             {:db/ident :age
+              :db/valueType :db.type/long
+              :db/cardinality :db.cardinality/one}])
+
+(def cfg {:store {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440001"} :initial-tx schema})
+
+(d/create-database cfg)
+(def conn (d/connect cfg))
+
+;; Query to find names and ages
+(def query '[:find ?n ?a :where [?e :name ?n] [?e :age ?a]])
+```
+
+## DB (Current State)
+
+`@conn` or `(d/db conn)` returns the current state - the most common view for queries:
+
+```clojure
+
+;; add first data
+(d/transact conn {:tx-data [{:name "Alice" :age 25}]})
+
+;; define simple query for name and age
+(def query '[:find ?n ?a :where [?e :name ?n] [?e :age ?a]])
+
+(d/q query @conn)
+;; => #{["Alice" 25]}
+
+;; update the entity
+(d/transact conn {:tx-data [{:db/id [:name "Alice"] :age 30}]})
+
+;; `db` reflects the latest state of the database
+(d/q query @conn)
+;; => #{["Alice" 30]}
+```
+
+## As-Of
+
+You can query the database at a specific point in time using `as-of`:
+
+```clojure
+(require '[datahike.api :as d])
+
+;; define simple schema
+(def schema [{:db/ident :name
+              :db/valueType :db.type/string
+              :db/unique :db.unique/identity
+              :db/index true
+              :db/cardinality :db.cardinality/one}
+             {:db/ident :age
+              :db/valueType :db.type/long
+              :db/cardinality :db.cardinality/one}])
+
+;; create our temporal database
+(def cfg {:store {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440003"} :initial-tx schema})
+
+(d/create-database cfg)
+
+(def conn (d/connect cfg))
+
+
+;; add first data
+(d/transact conn {:tx-data [{:name "Alice" :age 25}]})
+
+(def first-date (java.util.Date.))
+
+;; define simple query for name and age
+(def query '[:find ?n ?a :where [?e :name ?n] [?e :age ?a]])
+
+(d/q query  @conn)
+;; => #{["Alice" 25]}
+
+;; update the entity
+(d/transact conn {:tx-data [{:db/id [:name "Alice"] :age 30}]})
+
+;; let's compare the current and the as-of value:
+(d/q query  @conn)
+;; => #{["Alice" 30]}
+
+(d/q query (d/as-of @conn first-date))
+;; => #{["Alice" 25]}
+```
+
+## History
+
+For querying all data over the whole time span you may use `history` which joins
+current and all historical data:
+
+```clojure
+(require '[datahike.api :as d])
+
+;; define simple schema
+(def schema [{:db/ident :name
+              :db/valueType :db.type/string
+              :db/unique :db.unique/identity
+              :db/index true
+              :db/cardinality :db.cardinality/one}
+             {:db/ident :age
+              :db/valueType :db.type/long
+              :db/cardinality :db.cardinality/one}])
+
+;; create our temporal database
+(def cfg {:store {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440004"} :initial-tx schema})
+
+(d/create-database cfg)
+
+(def conn (d/connect cfg))
+
+;; add first data
+(d/transact conn {:tx-data [{:name "Alice" :age 25}]})
+
+;; define simple query for name and age
+(def query '[:find ?n ?a :where [?e :name ?n] [?e :age ?a]])
+
+;; history should have only one entry
+(d/q query (d/history @conn))
+;; => #{["Alice" 25]}
+
+;; update the entity
+(d/transact conn {:tx-data [{:db/id [:name "Alice"] :age 30}]})
+
+;; both entries are present
+(d/q query (d/history @conn))
+;; => #{["Alice" 30] ["Alice" 25]}
+```
+
+## Since
+
+Changes since a specific point in time can be searched by using the `since`
+database:
+
+```clojure
+(require '[datahike.api :as d])
+
+;; define simple schema
+(def schema [{:db/ident :name
+              :db/valueType :db.type/string
+              :db/unique :db.unique/identity
+              :db/index true
+              :db/cardinality :db.cardinality/one}
+             {:db/ident :age
+              :db/valueType :db.type/long
+              :db/cardinality :db.cardinality/one}])
+
+;; create our temporal database
+(def cfg {:store {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440005"} :initial-tx schema})
+
+(d/create-database cfg)
+
+(def conn (d/connect cfg))
+
+
+;; add first data
+(d/transact conn {:tx-data [{:name "Alice" :age 25}]})
+
+(def first-date (java.util.Date.))
+
+;; define simple query for name and age
+(def query '[:find ?n ?a :where [?e :name ?n] [?e :age ?a]])
+
+(d/q query @conn)
+;; => #{["Alice" 25]}
+
+;; update the entity
+(d/transact conn {:tx-data [{:db/id [:name "Alice"] :age 30}]})
+
+;; let's compare the current and the as-of value:
+(d/q query @conn)
+;; => #{["Alice" 30]}
+
+;; now we want to know any additions after a specific time
+(d/q query (d/since @conn first-date))
+;; => {}, because :name was transacted before the first date
+
+;; let's build a query where we use the latest db to find the name and the since db to find out who's age changed
+(d/q '[:find ?n ?a
+       :in $ $since
+       :where
+       [$ ?e :name ?n]
+       [$since ?e :age ?a]]
+     @conn
+     (d/since @conn first-date))
+```
+
+## Meta Entity
+
+With each transaction a meta entity is added to the index that stores the
+current point in time in the `:db/txInstant` attribute.
+
+With this data present in the current index, you can search and analyze them for
+your purposes.
+
+```clojure
+(require '[datahike.api :as d])
+
+;; define simple schema
+(def schema [{:db/ident :name
+              :db/valueType :db.type/string
+              :db/unique :db.unique/identity
+              :db/index true
+              :db/cardinality :db.cardinality/one}
+             {:db/ident :age
+              :db/valueType :db.type/long
+              :db/cardinality :db.cardinality/one}])
+
+;; create our temporal database
+(def cfg {:store {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440006"} :initial-tx schema})
+
+(d/create-database cfg)
+
+(def conn (d/connect cfg))
+
+;; add first data
+(d/transact conn {:tx-data [{:name "Alice" :age 25}]})
+
+;; let's find all transaction dates, should be two: one for the schema and one
+;; for the first data
+(d/q '[:find ?t :where [_ :db/txInstant ?t]] @conn)
+;; => #{[#inst "2019-08-16T11:40:28.794-00:00"] [#inst "2019-08-16T11:40:26.587-00:00"]}
+
+;; you might join over the tx id to get the date of any transaction
+(d/q '[:find ?n ?t :where [_ :name ?n ?tx] [?tx :db/txInstant ?t]] @conn)
+;; => #{["Alice" #inst "2019-08-16T11:40:28.794-00:00"]}
+```
+
+## Data Purging
+
+**Retraction vs Purging:** Normal retractions preserve data in history. Purging permanently deletes data from both current and historical indices.
+
+**When to purge:** Privacy regulations (GDPR, HIPAA, CCPA) requiring "right to deletion", sensitive data removal, or retention policy enforcement.
+
+**⚠️ Warning:** Purging is permanent and irreversible. Use only when legally required or explicitly needed.
+
+### Purge Operations
+
+- **`:db/purge`** - Remove specific datom (entity, attribute, value)
+- **`:db.purge/attribute`** - Remove all values for an attribute on an entity
+- **`:db.purge/entity`** - Remove entire entity and all its attributes
+- **`:db.history.purge/before`** - Remove all historical data before a date (retention policy cleanup)
+
+```clojure
+(require '[datahike.api :as d])
+
+;; define simple schema
+(def schema [{:db/ident :name
+              :db/valueType :db.type/string
+              :db/unique :db.unique/identity
+              :db/index true
+              :db/cardinality :db.cardinality/one}
+             {:db/ident :age
+              :db/valueType :db.type/long
+              :db/cardinality :db.cardinality/one}])
+
+;; create our temporal database
+(def cfg {:store {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440007"} :initial-tx schema})
+
+(d/create-database cfg)
+
+(def conn (d/connect cfg))
+
+
+;; add data
+(d/transact conn {:tx-data [{:name "Alice" :age 25}]})
+
+;; define simple query for name and age
+(def query '[:find ?n ?a :where [?e :name ?n] [?e :age ?a]])
+
+(d/q query  @conn)
+;; => #{["Alice" 25]}
+
+(d/transact conn {:tx-data [[:db.purge/entity [:name "Alice"]]]})
+
+;; data was removed from current database view
+(d/q query  @conn)
+;; => #{}
+
+;; data was also removed from history
+(d/q query (d/history @conn))
+;; => #{}
+```
+
+**Requirements for purging:**
+- History must be enabled (`:keep-history? true`)
+- Cannot purge schema attributes
+- Use retractions for normal data lifecycle - reserve purging for compliance requirements

package/doc/unstructured.md

@@ -0,0 +1,167 @@
+# Unstructured Input Support
+
+This experimental feature enables automatic schema inference from unstructured data (like JSON/EDN), allowing you to store richly nested data in Datahike with minimal setup.
+
+## Overview
+
+The unstructured input feature automatically:
+
+- Infers schema definitions from your data structure
+- Handles nested objects, collections, and primitive values
+- Converts complex data structures to Datahike's entity model
+- Works with both schema-on-read and schema-on-write configurations
+
+## Basic Usage
+
+```clojure
+(require '[datahike.api :as d])
+(require '[datahike.experimental.unstructured :as du])
+
+;; Initialize database (schema-on-read is best for unstructured data)
+(def cfg {:store {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440000"}
+          :schema-flexibility :read})
+(d/create-database cfg)
+(def conn (d/connect cfg))
+
+;; Insert complex unstructured data with a single function call
+(du/transact-unstructured conn 
+                        {:name "Alice"
+                         :age 42
+                         :addresses [{:city "New York" :primary true}
+                                     {:city "London" :primary false}]
+                         :skills ["programming" "design"]})
+```
+
+## Type and Cardinality Mapping
+
+The feature maps runtime values to Datahike types according to this conversion:
+
+| Input Type | Datahike Type | Cardinality |
+|------------|---------------|-------------|
+| Integer | `:db.type/long` | `:db.cardinality/one` |
+| Float | `:db.type/float` | `:db.cardinality/one` |
+| Double | `:db.type/double` | `:db.cardinality/one` |
+| String | `:db.type/string` | `:db.cardinality/one` |
+| Boolean | `:db.type/boolean` | `:db.cardinality/one` |
+| Keyword | `:db.type/keyword` | `:db.cardinality/one` |
+| Symbol | `:db.type/symbol` | `:db.cardinality/one` |
+| UUID | `:db.type/uuid` | `:db.cardinality/one` |
+| Date | `:db.type/instant` | `:db.cardinality/one` |
+| Map | `:db.type/ref` | `:db.cardinality/one` |
+| Vector/List (with values) | Based on first value | `:db.cardinality/many` |
+| Empty Vector/List | No schema generated until values exist | - |
+
+### Array Handling and Cardinality-Many
+
+When you provide an array/vector value, the system automatically:
+
+1. Infers the element type from the first item in the collection 
+2. Sets the attribute's cardinality to `:db.cardinality/many`
+3. Stores each value in the array as a separate datom
+
+This means you can model one-to-many relationships naturally:
+
+```clojure
+;; This will create tags with cardinality/many
+(du/transact-unstructured conn 
+                          {:name "Alice"
+                           :tags ["clojure" "database" "datalog"]})
+
+;; Later you can add more tags without losing existing ones
+(d/transact conn [{:db/id [:name "Alice"] 
+                   :tags ["awesome"]}])
+
+;; All tags will be preserved
+;; => ["clojure" "database" "datalog" "awesome"]
+```
+
+## Advanced Usage
+
+### Working with the Schema Directly
+
+If you need more control, you can process the data first to examine the schema:
+
+```clojure
+;; Process data to get both schema and transaction data
+(def result (du/process-unstructured-data 
+              {:user {:name "Bob" 
+                      :roles ["admin" "user"]}}))
+
+;; Examine the inferred schema
+(println (:schema result))
+;; => [{:db/ident :user, :db/valueType :db.type/ref, :db/cardinality :db.cardinality/one}
+;;     {:db/ident :name, :db/valueType :db.type/string, :db/cardinality :db.cardinality/one}
+;;     {:db/ident :roles, :db/valueType :db.type/string, :db/cardinality :db.cardinality/many}]
+
+;; Manually transact the schema first, if needed
+(d/transact conn (:schema result))
+
+;; Then transact the data
+(d/transact conn (:tx-data result))
+```
+
+### Schema-on-Write Databases
+
+For schema-on-write databases (the default), the feature automatically:
+
+1. Infers schema definitions from your data
+2. Adds any missing schema attributes in the same transaction
+3. Validates compatibility with existing schema
+4. Throws helpful errors if there's a conflict
+
+This makes the API particularly powerful for schema-on-write - you get the benefits of schema validation without the manual work of defining schemas upfront.
+
+```clojure
+;; With a schema-on-write database
+(def cfg {:store {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440001"}})
+(d/create-database cfg) 
+(def conn (d/connect cfg))
+
+;; This will add schema and data in a single transaction
+(du/transact-unstructured conn {:name "Charlie" :age 35})
+
+;; Later, you can add entities with new attributes
+;; The schema will automatically be extended
+(du/transact-unstructured conn {:name "Diana" 
+                                :age 28 
+                                :email "diana@example.com"}) ;; New attribute!
+```
+
+The feature supports incremental schema evolution - as your data model grows, the schema grows with it.
+
+### Schema Conflicts and Error Handling
+
+When using `transact-unstructured` with schema-on-write databases, the feature automatically detects schema conflicts. For example, if you've defined `:score` as a number but try to insert it as a string:
+
+```clojure
+;; First, establish schema for score as a number
+(d/transact conn [{:db/ident :score
+                   :db/valueType :db.type/long
+                   :db/cardinality :db.cardinality/one}])
+
+;; Add a valid score
+(d/transact conn [{:score 100}])
+
+;; This will fail with a schema conflict error:
+(du/transact-unstructured conn {:name "Bob", :score "High"})
+;; => ExceptionInfo: Schema conflict detected with existing database schema
+;;    {:conflicts [{:attr :score, :conflict :value-type, 
+;;                 :existing :db.type/long, :inferred :db.type/string}]}
+```
+
+The error messages provide details about the specific conflicts, making it easy to diagnose and fix issues.
+
+## Future Directions
+
+This experimental feature will likely be enhanced with:
+
+- Better handling of schema conflicts
+- Schema upgrading for schema-on-write databases 
+- Performance optimizations for large datasets
+- Special handling for additional data types
+
+## Limitations
+
+- Complex querying across nested entities requires knowledge of how the data was transformed
+- For schema-on-write, existing schema attributes must be compatible
+- Very large nested data structures might require chunking (not yet implemented)