datahike-browser-tests 1.0.0

package/doc/versioning.md

@@ -0,0 +1,261 @@
+# Versioning: Branch Databases, Not Just Code
+
+**Status: Beta - API becoming stable. Please try it out and provide feedback at [contact@datahike.io](mailto:contact@datahike.io).**
+
+Datahike's versioning system brings **git-like branching and merging** to your database. Just as git lets you experiment with code changes in branches before merging, Datahike lets you branch entire databases, evolve them independently, and selectively merge changes back.
+
+## Why Branch Databases?
+
+**Structural sharing makes branching efficient.** Unlike copying entire databases, Datahike branches share unchanged data through persistent data structures—the same principle that makes git fast. Creating a branch is nearly instantaneous regardless of database size, because only new or modified index nodes are written.
+
+When you create a branch, you get:
+- **Isolated evolution**: Experiment without affecting production data
+- **Selective merging**: Choose exactly which changes to apply
+- **Zero data duplication**: Shared data exists only once in storage
+- **Git-like semantics**: Branch, commit, merge with familiar concepts
+
+## When to Use Branches vs. Separate Databases
+
+**Use branches when:**
+- Testing schema migrations before applying to production
+- Creating staging environments for data review and approval
+- Running what-if analyses or experiments
+- Collaborative editing where changes need review before merging
+- You need to evolve a single logical dataset through different paths
+
+**Use separate databases when:**
+- Data is logically independent (different customers, different projects)
+- You want complete isolation with no shared storage
+- Combining data from multiple sources (data federation)
+- Scaling reads across completely separate datasets
+
+## How Branches Work with Distributed Index Space
+
+Branches are implemented as **different root pointers** in the same storage backend. Each branch name (`:db`, `:staging`, `:experimental`) points to a different commit, but all branches share the underlying persistent indices.
+
+This means:
+- Multiple readers can access different branches simultaneously via [DIS](distributed.md)
+- No coordination needed between readers on different branches
+- Branches can be accessed from any process with storage access
+- Each branch maintains its own transaction history
+
+## Relationship to Time-Travel Queries
+
+Datahike provides two complementary ways to work with history:
+
+**Versioning (branches):**
+- Creates durable, named snapshots that evolve independently
+- Allows merging changes between snapshots
+- Permanent until explicitly deleted
+- Use for: experiments, staging, alternative versions
+
+**Time-travel queries (as-of, history, since):**
+- Views past states of a single branch
+- Read-only access to transaction history
+- Automatic if `:keep-history? true`
+- Use for: auditing, debugging, temporal queries
+
+Both rely on the same persistent data structures, but serve different purposes.
+
+## API Overview
+
+The versioning API provides the following operations:
+
+- `branch!` - Create a new branch from an existing branch
+- `merge!` - Merge changes from one or more branches
+- `force-branch!` - Create a branch from any in-memory DB value
+- `delete-branch!` - Remove a branch
+- `branch-history` - View commit history for a branch
+- `commit-as-db` - Load a specific commit as a DB value
+- `branch-as-db` - Load the current state of a branch
+- `parent-commit-ids` - Get parent commits (for merge commits)
+
+All operations work with the connection's configured storage backend—no special setup required.
+
+## Example Use Cases
+
+### Testing Schema Migrations
+
+```clojure
+(require '[datahike.api :as d]
+         '[datahike.experimental.versioning :refer [branch! merge! delete-branch!]])
+
+(let [cfg {:store {:backend :file :path "/var/db/production"}
+           :keep-history? true
+           :schema-flexibility :write}
+      conn (d/connect cfg)]
+
+  ;; Create migration test branch
+  (branch! conn :db :migration-test)
+  (let [test-conn (d/connect (assoc cfg :branch :migration-test))]
+
+    ;; Try new schema
+    (d/transact test-conn [{:db/ident :email
+                            :db/valueType :db.type/string
+                            :db/cardinality :db.cardinality/one
+                            :db/unique :db.unique/identity}])
+
+    ;; Test with sample data
+    (d/transact test-conn [{:email "test@example.com"}])
+
+    ;; Verify migration worked, then merge to production
+    (when (verify-migration test-conn)
+      (merge! conn #{:migration-test} (migration-tx-data test-conn))
+      (delete-branch! conn :migration-test))))
+```
+
+### Staging Environment for Data Review
+
+```clojure
+;; Editorial workflow: draft changes in staging, review, then publish
+
+(let [cfg {:store {:backend :s3 :bucket "my-content-db"}
+           :schema-flexibility :write}
+      prod-conn (d/connect cfg)]
+
+  ;; Editor creates staging branch
+  (branch! prod-conn :db :staging)
+  (let [staging-conn (d/connect (assoc cfg :branch :staging))]
+
+    ;; Make draft changes
+    (d/transact staging-conn [{:article/title "New Article"
+                               :article/status :draft
+                               :article/content "..."}])
+
+    ;; Reviewers can read staging branch without affecting production
+    ;; ... review process ...
+
+    ;; Approved? Merge to production
+    (let [approved-changes (extract-approved-changes staging-conn)]
+      (merge! prod-conn #{:staging} approved-changes))))
+```
+
+### Running Experiments
+
+```clojure
+;; Test different recommendation algorithms without affecting live data
+
+(let [cfg {:store {:backend :file :path "/var/db/recommendations"}
+           :keep-history? false} ;; Don't need history for experiments
+      conn (d/connect cfg)]
+
+  ;; Create experimental branch
+  (branch! conn :db :experiment-new-algo)
+  (let [exp-conn (d/connect (assoc cfg :branch :experiment-new-algo))]
+
+    ;; Load experimental algorithm results
+    (d/transact exp-conn experimental-recommendations)
+
+    ;; Analyze results
+    (let [metrics (analyze-recommendations @exp-conn)]
+      (if (better-than-baseline? metrics)
+        ;; Good results - merge to production
+        (merge! conn #{:experiment-new-algo} experimental-recommendations)
+        ;; Poor results - just delete the branch
+        (delete-branch! conn :experiment-new-algo)))))
+```
+
+## Complete API Example
+
+The following example demonstrates the full versioning API:
+
+~~~clojure
+(require '[superv.async :refer [<?? S]]
+         '[datahike.api :as d]
+         '[datahike.experimental.versioning :refer [branch! branch-history delete-branch! force-branch! merge!
+                                                    branch-as-db commit-as-db parent-commit-ids]])
+
+(let [cfg    {:store              {:backend :file
+                                   :path    "/tmp/dh-versioning-test"}
+              :keep-history?      true
+              :schema-flexibility :write
+              :index              :datahike.index/persistent-set}
+      conn   (do
+              (d/delete-database cfg)
+              (d/create-database cfg)
+              (d/connect cfg))
+      schema [{:db/ident       :age
+               :db/cardinality :db.cardinality/one
+               :db/valueType   :db.type/long}]
+      _      (d/transact conn schema)
+      store  (:store @conn)]
+  (branch! conn :db :foo) ;; new branch :foo, does not create new commit, just copies
+  (let [foo-conn (d/connect (assoc cfg :branch :foo))] ;; connect to it
+    (d/transact foo-conn [{:age 42}]) ;; transact some data
+    ;; extracted data from foo by query
+    ;; ...
+    ;; and decide to merge it into :db
+    (merge! conn #{:foo} [{:age 42}]))
+  (count (parent-commit-ids @conn)) ;; => 2, as :db got merged from :foo and :db
+  ;; check that the commit stored is the same db as conn
+  (= (commit-as-db store (commit-id @conn)) (branch-as-db store :db) @conn) ;; => true
+  (count (<?? S (branch-history conn))) ;; => 4 commits now on both branches
+  (force-branch! @conn :foo2 #{:foo}) ;; put whatever DB value you have created in memory
+  (delete-branch! conn :foo))
+~~~
+
+Here we create a database as usual, but then we create a branch `:foo`, write to
+it and then merge it back. A simple query to extract all data in transactable
+form that is in a `branch1` db but not in `branch2` is
+
+~~~clojure
+(d/q [:find ?db-add ?e ?a ?v ?t
+      :in $ $2 ?db-add
+      :where
+      [$ ?e ?a ?v ?t]
+      [(not= :db/txInstant ?a)]
+      (not [$2 ?e ?a ?v ?t])]
+      branch1 branch2 :db/add)
+~~~
+
+but you might want to be more selective when creating the data for `merge!`.
+
+## Query Pattern: Extracting Branch Differences
+
+When merging, you typically want to extract only specific changes from a branch. Here's a general pattern for finding differences:
+
+```clojure
+;; Find all datoms in branch1 that are not in branch2
+(defn branch-diff [branch1 branch2]
+  (d/q '[:find ?e ?a ?v ?t
+         :in $ $2
+         :where
+         [$ ?e ?a ?v ?t]
+         [(not= :db/txInstant ?a)]
+         (not [$2 ?e ?a ?v ?t])]
+       branch1 branch2))
+
+;; Extract as transaction data
+(defn diff-as-tx-data [branch1 branch2]
+  (mapv (fn [[e a v t]] [:db/add e a v])
+        (branch-diff branch1 branch2)))
+```
+
+You can extend this pattern to:
+- Filter by specific attributes (e.g., only user-facing data)
+- Extract only entities matching certain criteria
+- Apply transformations before merging
+- Validate changes before committing to the target branch
+
+## Integration with Existing Connections
+
+Branches integrate seamlessly with Datahike's connection model:
+
+```clojure
+;; Connect to specific branch by name
+(def staging-conn (d/connect {:store {...} :branch :staging}))
+
+;; Default branch is :db
+(def main-conn (d/connect {:store {...}})) ;; same as :branch :db
+
+;; Each connection operates independently
+(d/transact staging-conn [...])  ;; doesn't affect main-conn
+@staging-conn  ;; DB snapshot of :staging branch
+@main-conn     ;; DB snapshot of :db branch
+```
+
+Branches work with all storage backends (file, S3, JDBC, etc.) and participate in [Distributed Index Space](distributed.md)—multiple processes can read different branches concurrently.
+
+## Feedback and Support
+
+We are actively developing the versioning API and would love to hear about your use cases. If you have ideas, feature requests, or encounter issues, please reach out to [contact@datahike.io](mailto:contact@datahike.io) or open an issue on [GitHub](https://github.com/replikativ/datahike/issues).

package/examples/basic/README.md

@@ -0,0 +1,19 @@
+# Examples
+
+This project shows some use cases for the store, the schema, and temporal index implemented in datahike.
+
+## Usage
+
+Open in `/src/examples` the topic you want to explore in your favourite editor, use a clojure repl to
+eval the expressions from top to bottom. 
+
+For the [PostgreSQL](https://www.postgresql.org) example in `store.clj` you need to have 
+[docker](https://www.docker.com/) and
+[docker-compose](https://docs.docker.com/compose/) installed.
+Start it with:
+
+``` sh
+docker-compose up -d
+```
+
+If the selected ports collide with other ports, you may want to adjust `/docker-compose.yml.` and restart the container.

package/examples/basic/deps.edn

@@ -0,0 +1,6 @@
+{:paths ["src"]
+ :deps  {org.clojure/clojure         {:mvn/version "1.11.1"}
+         org.replikativ/datahike     {:local/root "../.."}
+         ;; For released versions, use:
+         ;; org.replikativ/datahike   {:mvn/version "0.7.1620"}
+         }}

package/examples/basic/docker-compose.yml

@@ -0,0 +1,13 @@
+version: '3.4'
+services:
+  db:
+    image: postgres
+    restart: always
+    environment:
+      POSTGRES_USER: datahike
+      POSTGRES_PASSWORD: clojure
+      POSTGRES_DB: "pg-example"
+    ports:
+      - "5437:5432"
+    volumes:
+      - /tmp/datahike_example/data:/var/lib/postgresql/data

package/examples/basic/src/examples/core.clj

@@ -0,0 +1,60 @@
+(ns examples.core
+  (:require [datahike.api :as d]))
+
+(def cfg {:store {:backend :file
+                  :path "/tmp/example"}
+          :keep-history? true
+          :schema-flexibility :read})
+
+;; create a database at this place, per default configuration we enforce a strict
+;; schema and keep all historical data
+(d/create-database cfg)
+
+(def conn (d/connect cfg))
+
+;; the first transaction will be the schema we are using
+;; you may also add this within database creation by adding :initial-tx
+;; to the configuration
+(d/transact conn [{:db/ident :name
+                   :db/valueType :db.type/string
+                   :db/cardinality :db.cardinality/one }
+                  {:db/ident :age
+                   :db/valueType :db.type/long
+                   :db/cardinality :db.cardinality/one }])
+
+;; lets add some data and wait for the transaction
+(d/transact conn [{:name  "Alice", :age   20 }
+                  {:name  "Bob", :age   30 }
+                  {:name  "Charlie", :age   40 }
+                  {:age 15 }])
+
+;; search the data
+(d/q '[:find ?e ?n ?a
+       :where
+       [?e :name ?n]
+       [?e :age ?a]]
+  @conn)
+;; => #{[3 "Alice" 20] [4 "Bob" 30] [5 "Charlie" 40]}
+
+;; add new entity data using a hash map
+(d/transact conn {:tx-data [{:db/id 3 :age 25}]})
+
+;; if you want to work with queries like in
+;; https://grishaev.me/en/datomic-query/,
+;; you may use a hashmap
+(d/q {:query '{:find [?e ?n ?a ]
+               :where [[?e :name ?n]
+                       [?e :age ?a]]}
+      :args [@conn]})
+;; => #{[5 "Charlie" 40] [4 "Bob" 30] [3 "Alice" 25]}
+
+;; query the history of the data
+(d/q '[:find ?a
+       :where
+       [?e :name "Alice"]
+       [?e :age ?a]]
+  (d/history @conn))
+;; => #{[20] [25]}
+
+;; clean up the database if it is not needed any more
+(d/delete-database cfg)

package/examples/basic/src/examples/schema.clj

@@ -0,0 +1,155 @@
+(ns examples.schema
+  (:require [datahike.api :as d]))
+
+;; The first example assumes you know your data model in advanve,
+;; so you we can use a schema-on-write approach in contrast to a schema-on-read
+;; approach. Have a look at the documentation in `/doc/schema.md` for more
+;; information on the different types of schema flexibility. After the first
+;; example we will have a short schema-on-read example.
+
+; first define data model
+(def schema [{:db/ident :contributor/name
+              :db/valueType :db.type/string
+              :db/unique :db.unique/identity
+              :db/index true
+              :db/cardinality :db.cardinality/one
+              :db/doc "a contributor's name"}
+             {:db/ident :contributor/email
+              :db/valueType :db.type/string
+              :db/cardinality :db.cardinality/many
+              :db/doc "a contributor's email"}
+             {:db/ident :repository/name
+              :db/valueType :db.type/string
+              :db/unique :db.unique/identity
+              :db/index true
+              :db/cardinality :db.cardinality/one
+              :db/doc "a repository's name"}
+             {:db/ident :repository/contributors
+              :db/valueType :db.type/ref
+              :db/cardinality :db.cardinality/many
+              :db/doc "the repository's contributors"}
+             {:db/ident :repository/public
+              :db/valueType :db.type/boolean
+              :db/cardinality :db.cardinality/one
+              :db/doc "toggle whether the repository is public"}
+             {:db/ident :repository/tags
+              :db/valueType :db.type/ref
+              :db/cardinality :db.cardinality/many
+              :db/doc "the repository's tags"}
+             {:db/ident :language/clojure}
+             {:db/ident :language/rust}])
+
+;; define configuration
+(def cfg {:store {:backend :mem
+                  :id "schema-intro"}
+          :schema-flexibility :write})
+
+;; cleanup previous database
+(d/delete-database cfg)
+
+;; create the in-memory database
+(d/create-database cfg)
+
+;; connect to it
+(def conn (d/connect cfg))
+
+;; add the schema
+
+(d/transact conn schema)
+
+;; let's insert our first user
+(d/transact conn [{:contributor/name "alice" :contributor/email "alice@exam.ple"}])
+
+;; let's find her with a query
+(def find-name-email '[:find ?e ?n ?em :where [?e :contributor/name ?n] [?e :contributor/email ?em]])
+
+(d/q find-name-email @conn)
+
+;; let's find her directly, as contributor/name is a unique, indexed identity
+(d/pull @conn '[*] [:contributor/name "alice"])
+
+;; add a second email, as we have a many cardinality, we can have several ones as a user
+(d/transact conn [{:db/id [:contributor/name "alice"] :contributor/email "alice@test.test"}])
+
+;; let's see both emails
+(d/q find-name-email @conn)
+
+;; try to add something completely not defined in the schema
+(d/transact conn [{:something "different"}])
+;; => Exception shows missing schema definition
+
+;; try to add wrong contributor values
+(d/transact conn [{:contributor/email :alice}])
+;; => Exception shows what value is expected
+
+;; add another contributor by using a the alternative transaction schema that expects a hash map with tx-data attribute
+(d/transact conn {:tx-data [{:contributor/name "bob" :contributor/email "bob@ac.me"}]})
+
+(d/q find-name-email @conn)
+
+(d/pull @conn '[*] [:contributor/name "bob"])
+
+;; change bob's name to bobby
+(d/transact conn [{:db/id [:contributor/name "bob"] :contributor/name "bobby"}])
+
+;; check it
+(d/q find-name-email @conn)
+
+(d/pull @conn '[*] [:contributor/name "bobby"])
+
+;; bob is not related anymore as index
+(d/pull @conn '[*] [:contributor/name "bob"])
+;; will give an exception
+
+;; create a repository, with refs from uniques, and an ident as enum
+(d/transact conn [{:repository/name "top secret"
+                   :repository/public false
+                   :repository/contributors [[:contributor/name "bobby"] [:contributor/name "alice"]]
+                   :repository/tags :language/clojure}])
+
+;; let's search with pull inside the query
+(def find-repositories '[:find (pull ?e [*]) :where [?e :repository/name ?n]])
+
+;; looks good
+(d/q find-repositories @conn)
+
+;; let's go further and fetch the related contributor data as well
+(def find-repositories-with-contributors '[:find (pull ?e [* {:repository/contributors [*] :repository/tags [*]}]) :where [?e :repository/name ?n]])
+
+(d/q find-repositories-with-contributors @conn)
+
+;; the schema is part of the index, so we can query them too.
+;; Let's find all attribute names and their description.
+(d/q '[:find ?a ?d :where [?e :db/ident ?a] [?e :db/doc ?d]] @conn)
+
+;; cleanup the database
+(d/delete-database cfg)
+
+;; Schema On Read
+
+;; let's create another database that can hold any arbitrary data
+
+(def cfg {:store {:backend :mem
+                  :id "schemaless"}
+          :schema-flexibility :read})
+
+(d/create-database cfg)
+
+(def conn (d/connect cfg))
+
+;; now we can go wild and transact anything
+(d/transact conn [{:any "thing"}])
+
+;; use simple query on this data
+(d/q '[:find ?v :where [_ :any ?v]] @conn)
+
+;; be aware: although there is no schema, you should tell the database if some
+;; attributes can have specific cardinality or indices.
+;; You may add that as schema transactions like before
+(d/transact conn [{:db/ident :any :db/cardinality :db.cardinality/many}])
+
+;; let's add more data to the first any entity
+(def any-eid (d/q '[:find ?e . :where [?e :any "thing"]] @conn))
+(d/transact conn [{:db/id any-eid :any "thing else"}])
+
+(d/q '[:find ?v :where [_ :any ?v]] @conn)

package/examples/basic/src/examples/store.clj

@@ -0,0 +1,60 @@
+(ns examples.store
+  (:require [datahike.api :as d]))
+
+(def schema [{:db/ident :name
+              :db/valueType :db.type/string
+              :db/cardinality :db.cardinality/one}])
+
+(def query '[:find ?n :where [?e :name ?n]])
+
+;; let's cleanup, create, and connect all in one
+(defn cleanup-and-create-conn [cfg]
+  (d/delete-database cfg)
+  (d/create-database cfg)
+  (let [conn (d/connect cfg)]
+    (d/transact conn schema)
+    conn))
+
+(defn transact-and-find [conn name]
+  (d/transact conn [{:name name}])
+  (d/q query @conn))
+
+;; first let's have a look at the memory store which uses an atom internally to store data
+;; memory backend requires a UUID identifier for distributed tracking
+(def mem-cfg {:store {:backend :memory :id (java.util.UUID/randomUUID)}})
+
+;; create it
+(def mem-conn (cleanup-and-create-conn mem-cfg))
+
+;; add and find data
+(transact-and-find mem-conn "Alice");; => #{["Alice"]}
+
+;; next we try out file based store which can be used as the simplest form of persistence
+;; the datoms are serialized at `/tmp/file_example`
+(def file-cfg {:store {:backend :file :path "/tmp/file_example"}})
+
+(def file-conn (cleanup-and-create-conn file-cfg))
+
+(transact-and-find file-conn "Bob");; => #{["Bob"]}
+
+;; External backends
+;; Datahike supports additional backends via plugins:
+;; - PostgreSQL (via datahike-jdbc)
+;; - S3 (via datahike-s3)
+;; - Redis, LevelDB, etc.
+;; See https://github.com/replikativ/datahike for available backends
+
+;; We can query across multiple databases
+(d/q '[:find ?mem ?file
+       :in $mem-db $file-db
+       :where
+       [$mem-db ?e0 :name ?mem]
+       [$file-db ?e1 :name ?file]]
+     (d/db mem-conn)
+     (d/db file-conn)) ;; => #{["Alice" "Bob"]}
+
+;; cleanup
+(do
+  (d/delete-database mem-cfg)
+  (d/delete-database file-cfg))
+