How can changes to be made to a Datomic database be grouped in a way that they either all succeed or they all fail? Think of transactions in SQL, via BEGIN and COMMIT/ROLLBACK.
In a non-trivial system there are some high-level actions that require several small overlapping changes to the data. Having a subset of the changes go through and the rest fail is important to avoid.
I tried forking the db using d/with, then accumulating tx-data with the idea that I could transact them all at some later time. That strategy breaks if the tx-data has conflicting datoms, or involves retractions or other database functions. I also thought of just using d/with repeatedly but accumulating the :tx-data lists from the response of d/with, in order to avoid issues with database functions, but that of course means that the benefit of database functions (acting atomically on the latest db value) is lost, and it still doesn’t solve the issue of conflicting datoms.
Combining changes manually into a single d/transact call is not a valid solution because it implies that I need to write two copies of the same logic - one that is a function of entity->tx-data and one that is a function of tx-data->tx-data.
As you mention, like git branch merges, you are vulnerable to “merge conflicts” which are essential complexity and are your problem to resolve. But I dont know if we are talking about the same thing because in my version, :db.fn/retractEntity and transaction functions work (they will all evaluate again atomically in the transactor). What type of merge conflicts are you seeing? I dont think duplicate statements that assert the same thing are a problem, and upsert helps too. You may also need to deal with tempid reversing, but that can be handled inside of comp-step abstraction i think.
Hi Dustin, thanks for the idea. You seem to have a good handle on the problem.
I wrote something similar on Thu/Fri and found it worked for simple cases but not if I caused “conflicting” datoms. I was/am thinking of the case where maybe there’s a status field on an entity that gets set to state A but some other logic overrides it to state B, and because those two things happen in the same transaction they fail because of a perceived conflict.
I was also worried that tempid->entity-id replacements would be impossible to reverse if they were hidden inside a database function, and I wasn’t happy writing a critical layer of software without making it handle all cases. Your use of the term “merge conflict” is convincing me that the number of problematic cases is actually smaller than I had feared.
To Datomic authors/maintainers: why not chime in? I don’t accept that writing transaction merging is a developer’s responsibility. Having proper transactions, like in SQL, is a requirement for me to promote Datomic as a real-world tool in the future.
Can you demonstrate why the system must be factored like this in the first place? I would try to refactor it to this: (concat (step-1 $) (step-2 $) (step-3 $)) and if this can’t be done i am pretty keen to understand why not
A simple concat like that doesn’t play well with the map form (where concat would need to be replaced with a deep merge). Also it could fail at an inconvenient time. One reason people use Datomic is because it claims to have a solid foundation, and I wouldn’t want to layer a “works most of the time” solution on top. Yes, it’ll be fine for a few non-overlapping datoms in list form, but I expect that most people enjoy throwing nested maps at d/transact and never needing to use the list form.
As for a concrete example, it’s not relevant because I’m not trying to implement a specific transaction, I’m trying to make a system that allows for code reuse and doesn’t shoot my successor in the foot. I am asking for composable changes, that’s it, that’s all.
I understand that the transactions I’m asking for aren’t provided by Cognitect. I’ll implement them myself but with the feeling of doing the equivalent of writing homemade encryption
That being said, I don’t think Datomic should complain about “conflicting” datoms in the first place, which might be the real issue here; it should just do the reasonable thing and transact them anyhow, using the order of datoms passed to d/transact as the tiebreaker.
Yeah, it might be tricky to properly merge your transactions. I don’t think I would try to do that, although it should be possible.
Could you assign a unique identifier to all the Datomic transactions in the same “group” and have a function to revert all the txs in a given group? It won’t be atomic but I have used this method in the past when dealing with logical txs made of multiple actual Datomic txs.
@tar, transaction functions in Datomic are the “traditional transactions” you are looking for. The allow to group several operations, like read, modify, wite, into a transaction. Database Functions | Datomic
avodonosov, I was describing my attempt to merge multiple transactions before calling d/transact. Because function calls are opaque, it wouldn’t be possible to walk them and replace tempids as necessary. That being said, you have a good point, why not use db functions more, to the point where I have a large chunk of code in Datomic? No real reason beyond source control maybe, and transactor performance. But if I made the functions quick and pure, maybe it could work! Thanks, I’m going to see what I can do with that idea.
I am asking for composable changes, that’s it, that’s all.
I think you’ve asked for composable effects, which is the root of the cognitive dissonance here.
Transaction map sugar works with concat:
(concat
[{:db/id "a"
:order/lineItems [{:lineItem/product "chocolate"
:lineItem/quantity 1}]}]
[[:db/add "a" :order/lineItems "b"]
[:db/add "b" :lineItem/product "whisky"]
[:db/add "b" :lineItem/quantity 2]])
(The seattle schema similarly mixes vector and map form)
If the application code is generating collisions you can compensate here, for example transforming the transaction value to vector syntax and introducing a notion of statement ordering as you suggested above.
I dealt with the same issue. Dustin, your suggestion regarding introducing a notion of statement ordering is more or less what I ended up with.
I’m posting this here in the hopes that it’ll help out @tar and that someone can look at the solution and let me know whether it seems reasonable. Like tar said, I feel a bit like I’m writing homemade encryption.
Here’s my use case and the issues I ran into:
My server accepts “commands” that look like re-frame events. For example:
[:artist/create] [:artist/set-name artist-id "new name"]
Each command is mapped to a function that produces a sequence of facts, as could be passed to d/transact:
This works fine when handling a command at a time, but it is a requirement of my UI that there is an explicit save step and that all commands that happen between saves are saved at once and succeed or fail as a unit. Handling batches of commands reveals the conflicting datoms issue that has already been discussed and another problem I found a bit more vexing: tempids need to be consistent across the all the facts for a command batch. If we create an artist with :artist/create, then set that new artist’s name in one transaction, set-artist-name facts need to use the right tempid for the artist.
Ignoring the application-side code that handles making the fact-sequence-per-command, my solution was to add a transactor function that accepts any number of fact-sequences and converts them to a single fact sequence.
I pasted our datomic util lib into a gist (not putting it in a public repo just yet):
A lot of this code isn’t relevant to what we’re talking about. The part that matters is the last fn, transact-many-validated. I installed that as a transactor fn and do all my command-related transactions through that, like (d/transact [:kc.tx-fns/transact-many-validated sequence-of-fact-sequences]). The heart of the implementation is in -simplify-fact-seqs.
You could make your own transaction fn that doesn’t have our special FK and retraction logic - it would look like:
(defn transact-many-validated
"Given any number of fact sequences, reduce them to a single fact sequence without any conflicting datoms."
[db fact-seqs]
(let [fact-seqs (map clean-facts fact-seqs)
fact-seqs (simplify-fact-seqs db fact-seqs)
facts (merge-facts db fact-seqs)
facts (remove-tempid-retractions facts)]
facts))
For tempid reversing: Hyperfiddle maintains a branch stack (cascading d/withs). Views always see the reified ids, consistent with the time-basis of the result. Branches are a datomic transaction value, and are stored with the top layer of tempids reversed. When a branch is merged into parent (in response to a user gesture), we 1) reverse the next layer of tempids, 2) “concat” with into-tx. Since we never need to look at reified tx entities – we always hang on to the transaction as specified by the programmer/user – db fns don’t need to be accounted for. Everything is pretty orderly and idiomatic.
