Traditional transactions (commit multiple changes at once)


#1

How can changes to be made to a Datomic database be grouped in a way that they either all succeed or they all fail? Think of transactions in SQL, via BEGIN and COMMIT/ROLLBACK.

In a non-trivial system there are some high-level actions that require several small overlapping changes to the data. Having a subset of the changes go through and the rest fail is important to avoid.

I tried forking the db using d/with, then accumulating tx-data with the idea that I could transact them all at some later time. That strategy breaks if the tx-data has conflicting datoms, or involves retractions or other database functions. I also thought of just using d/with repeatedly but accumulating the :tx-data lists from the response of d/with, in order to avoid issues with database functions, but that of course means that the benefit of database functions (acting atomically on the latest db value) is lost, and it still doesn’t solve the issue of conflicting datoms.

Combining changes manually into a single d/transact call is not a valid solution because it implies that I need to write two copies of the same logic - one that is a function of entity->tx-data and one that is a function of tx-data->tx-data.


#2

Does this accomplish what you want without intermediate effects?

(defn step-1 [$]
  ; Each step can query as if the transactions went through
  ; each step must return a dbval for next guy
  [$ [[:db/add "a" :todo/title "buy groceries"]]])

(defn step-2 [$]
  [$ [[:db/add "b" :todo/title "feed baby"]]])

(defn step-3 [$]
  [$ [[:db/add "a" :todo/completed true]]])


(comment
  (def $ (d/db (d/connect "datomic:free://datomic:4334/~dustin.getz")))
  (let [[$ tx] (step-1 $)
        [$ tx'] (step-2 (:db-after (d/with $ tx)))
        [$ tx''] (step-3 (:db-after (d/with $ tx')))]
    [$ (concat tx tx' tx'')])

  #_[datomic.db.Db @e8096e87
     ([:db/add "a" :todo/title "buy groceries"]
       [:db/add "b" :todo/title "feed baby"]
       [:db/add "a" :todo/completed true])]
  )

(defn comp-step [f g]
  (fn [$]
    (let [[$ tx] (g $)
          [$ tx'] (f (:db-after (d/with $ tx)))]
      [$ (concat tx tx')])))

(comment

  ((comp-step step-2 step-1) $)
  #_[datomic.db.Db @de8957c1
     ([:db/add "a" :todo/title "buy groceries"]
       [:db/add "b" :todo/title "feed baby"])]

  ((comp-step step-3 (comp-step step-2 step-1)) $)
  #_[datomic.db.Db @2d470e30
     ([:db/add "a" :todo/title "buy groceries"]
       [:db/add "b" :todo/title "feed baby"]
       [:db/add "a" :todo/completed true])]

  ((reduce comp-step [step-3 step-2 step-1]) $)
  #_[datomic.db.Db @75af774c
     ([:db/add "a" :todo/title "buy groceries"]
       [:db/add "b" :todo/title "feed baby"]
       [:db/add "a" :todo/completed true])]

  )

#3

As you mention, like git branch merges, you are vulnerable to “merge conflicts” which are essential complexity and are your problem to resolve. But I dont know if we are talking about the same thing because in my version, :db.fn/retractEntity and transaction functions work (they will all evaluate again atomically in the transactor). What type of merge conflicts are you seeing? I dont think duplicate statements that assert the same thing are a problem, and upsert helps too. You may also need to deal with tempid reversing, but that can be handled inside of comp-step abstraction i think.


#4

Hi Dustin, thanks for the idea. You seem to have a good handle on the problem.

I wrote something similar on Thu/Fri and found it worked for simple cases but not if I caused “conflicting” datoms. I was/am thinking of the case where maybe there’s a status field on an entity that gets set to state A but some other logic overrides it to state B, and because those two things happen in the same transaction they fail because of a perceived conflict.

I was also worried that tempid->entity-id replacements would be impossible to reverse if they were hidden inside a database function, and I wasn’t happy writing a critical layer of software without making it handle all cases. Your use of the term “merge conflict” is convincing me that the number of problematic cases is actually smaller than I had feared.

To Datomic authors/maintainers: why not chime in? I don’t accept that writing transaction merging is a developer’s responsibility. Having proper transactions, like in SQL, is a requirement for me to promote Datomic as a real-world tool in the future.


#5

Can you demonstrate why the system must be factored like this in the first place? I would try to refactor it to this: (concat (step-1 $) (step-2 $) (step-3 $)) and if this can’t be done i am pretty keen to understand why not


#6

A simple concat like that doesn’t play well with the map form (where concat would need to be replaced with a deep merge). Also it could fail at an inconvenient time. One reason people use Datomic is because it claims to have a solid foundation, and I wouldn’t want to layer a “works most of the time” solution on top. Yes, it’ll be fine for a few non-overlapping datoms in list form, but I expect that most people enjoy throwing nested maps at d/transact and never needing to use the list form.

As for a concrete example, it’s not relevant because I’m not trying to implement a specific transaction, I’m trying to make a system that allows for code reuse and doesn’t shoot my successor in the foot. I am asking for composable changes, that’s it, that’s all.

I understand that the transactions I’m asking for aren’t provided by Cognitect. I’ll implement them myself but with the feeling of doing the equivalent of writing homemade encryption :slight_smile:

That being said, I don’t think Datomic should complain about “conflicting” datoms in the first place, which might be the real issue here; it should just do the reasonable thing and transact them anyhow, using the order of datoms passed to d/transact as the tiebreaker.


#7

Yeah, it might be tricky to properly merge your transactions. I don’t think I would try to do that, although it should be possible.

Could you assign a unique identifier to all the Datomic transactions in the same “group” and have a function to revert all the txs in a given group? It won’t be atomic but I have used this method in the past when dealing with logical txs made of multiple actual Datomic txs.


#8

@tar, transaction functions in Datomic are the “traditional transactions” you are looking for. The allow to group several operations, like read, modify, wite, into a transaction. https://docs.datomic.com/on-prem/database-functions.html

Why don’t you use them?

in order to avoid issues with database functions

What issues?


#9

benfle, that could work, and it’s pretty low effort. Thanks.


#10

avodonosov, I was describing my attempt to merge multiple transactions before calling d/transact. Because function calls are opaque, it wouldn’t be possible to walk them and replace tempids as necessary. That being said, you have a good point, why not use db functions more, to the point where I have a large chunk of code in Datomic? No real reason beyond source control maybe, and transactor performance. But if I made the functions quick and pure, maybe it could work! Thanks, I’m going to see what I can do with that idea.


#11

I am asking for composable changes, that’s it, that’s all.

I think you’ve asked for composable effects, which is the root of the cognitive dissonance here.

Transaction map sugar works with concat:

(concat
  [{:db/id "a"
    :order/lineItems [{:lineItem/product "chocolate"
                       :lineItem/quantity 1}]}]
  [[:db/add "a" :order/lineItems "b"]
   [:db/add "b" :lineItem/product "whisky"]
   [:db/add "b" :lineItem/quantity 2]])

(The seattle schema similarly mixes vector and map form)

If the application code is generating collisions you can compensate here, for example transforming the transaction value to vector syntax and introducing a notion of statement ordering as you suggested above.