Upsert behavior with composite tuple key

Datomic Cloud
1

Although the composite tuple key does get added on the second transaction, the third transaction fails. A transacted entity won’t actually resolve to an existing entity unless I specify the :foo+bar key explicitly, which seems like a bug (or at least undesirable behavior). So this means that if you want upsert behavior, you have to include both the composite attribute and the individual attributes whenever you transact an entity.

user=> 
(let [db (d/with-db system/conn)
        txes [[#:db{:ident :foo,
                    :valueType :db.type/string,
                    :cardinality :db.cardinality/one}
               #:db{:ident :bar,
                    :valueType :db.type/string,
                    :cardinality :db.cardinality/one}
               #:db{:ident :baz,
                    :valueType :db.type/string,
                    :cardinality :db.cardinality/one}
               {:db/ident :foo+bar,
                :db/cardinality :db.cardinality/one
                :db/unique :db.unique/identity
                :db/valueType :db.type/tuple
                :db/tupleAttrs [:foo :bar]}]
              [{:foo "foo"
                :bar "bar"}]
              ; doesn't work
              [{:foo "foo"
                :bar "bar"
                :baz "baz"}]
              ; works
              [{:foo "foo"
                :bar "bar"
                :foo+bar ["foo" "bar"]
                :baz "baz"}]]]
    (reduce (fn [db tx]
              (try
                (let [db (:db-after (d/with db {:tx-data tx}))]
                  (println "transaction successful")
                  db)
                (catch Exception e
                  (println "transaction failed:" (.getMessage e))
                  db)))
            db txes))
transaction successful
transaction successful
transaction failed: Unique conflict: :foo+bar, value: ["foo" "bar"] already held by: 51263630133428781 asserted for: 4631142976193070
transaction successful

(Also, I noticed that the schema data reference still says:

Datomic does not provide a mechanism to declare composite uniqueness constraints; however, you can implement them (or any arbitrary functional constraint) via transaction functions.

This should probably be updated to mention composite tuples.)

6 Likes
2

Update: upsert is completely broken when the key tuple includes a false boolean value. Example from my application:

(d/transact sys/conn
            {:tx-data [{:rec.oc/track 9895604649984119
                        :rec.oc/user 28666467159376431
                        :rec.oc/hit? false
                        :rec.oc/key [28666467159376431 9895604649984119 false]}]})
Unique conflict: :rec.oc/key, value: [28666467159376431 9895604649984119 false] already held by: 18159534044352573 asserted for: 4925812092440654

I’m assuming this is related to this. Notes:

  • False values only showed up as nil when using d/with, not when doing a real transaction (note value: [28666467159376431 9895604649984119 false] in the example above).
  • The above example works when using d/with, it seems that upsert-for-keys-with-false-values is only completely broken for real transactions (at least when using d/with it’s only partially broken :wink: ).

For now I’m thinking I’ll have to just use another attribute in place of booleans, e.g. a keyword attribute with :true and :false as the values.

3

Edit: did some more investigating.

  (d/create-database client {:db-name "test"})
  (let [conn (d/connect client {:db-name "test"})
        txes [[#:db{:ident :foo,
                    :valueType :db.type/string,
                    :cardinality :db.cardinality/one}
               #:db{:ident :bar,
                    :valueType :db.type/boolean,
                    :cardinality :db.cardinality/one}
               #:db{:ident :baz,
                    :valueType :db.type/string,
                    :cardinality :db.cardinality/one}
               {:db/ident :foo+bar,
                :db/cardinality :db.cardinality/one
                :db/unique :db.unique/identity
                :db/valueType :db.type/tuple
                :db/tupleAttrs [:foo :bar]}]

              ; The value of :foo+bar after this transaction will be ["a" nil]:
              [{:foo "a"
                :bar false}]

              ; And somehow that causes this to fail:
              [{:foo "a"
                :bar false}]

              ; Upsert will work on the first time if we set :foo+bar explicitly.
              ; Also notice that :foo+bar is actually set to ["a" false] after this transaction.
              [{:foo "a"
                :bar false
                :foo+bar ["a" false]}]

              ; But on the second time, it'll create a new entity with :foo+bar set to ["a" nil]
              [{:foo "a"
                :bar false
                :foo+bar ["a" false]}]

              ; If we now try to do it again, changing the second entity to ["a" false] will fail
              ; because one already exists:
              [{:foo "a"
                :bar false
                :foo+bar ["a" false]}]]]
    (doseq [tx txes]
      (try
        (let [db (:db-after (d/transact conn {:tx-data tx}))]
          (println "transaction:")
          (pprint tx)
          (println "result:")
          (pprint
            (d/q '[:find (pull ?e [*]) :where [?e :foo]]
                 db))
          (println))
        (catch Exception e
          (println "transaction failed:" (.getMessage e))
          (println)))))
transaction:                                                                                                                                                                                                                         
[#:db{:ident :foo,
      :valueType :db.type/string,
      :cardinality :db.cardinality/one}
 #:db{:ident :bar,
      :valueType :db.type/boolean,
      :cardinality :db.cardinality/one}
 #:db{:ident :baz,
      :valueType :db.type/string,
      :cardinality :db.cardinality/one}
 #:db{:ident :foo+bar,
      :cardinality :db.cardinality/one,
      :unique :db.unique/identity,
      :valueType :db.type/tuple,
      :tupleAttrs [:foo :bar]}]
result:
[]

transaction:
[{:foo "a", :bar false}]
result:
[[{:db/id 53137197947158605, :foo "a", :bar false, :foo+bar ["a" nil]}]]

transaction failed: Unique conflict: :foo+bar, value: ["a" nil] already held by: 53137197947158605 asserted for: 69475940735909966

transaction:
[{:foo "a", :bar false, :foo+bar ["a" false]}]
result:
[[{:db/id 53137197947158605,
   :foo "a",
   :bar false,
   :foo+bar ["a" false]}]]

transaction:
[{:foo "a", :bar false, :foo+bar ["a" false]}]
result:
[[{:db/id 53137197947158605,
   :foo "a",
   :bar false,
   :foo+bar ["a" false]}]
 [{:db/id 19624083532546126, :foo "a", :bar false, :foo+bar ["a" nil]}]]

transaction failed: Unique conflict: :foo+bar, value: ["a" false] already held by: 53137197947158605 asserted for: 19624083532546126

So, in any case, it does seem that composite tuple handling of false values is badly broken and they (i.e., composite tuples with boolean values) shouldn’t be used until this is fixed.

4

Hi @jacob thank you for bringing this issue to our attention. We have identified an issue with the treatment of :false in tuples and are working on a fix to address the issue in an upcoming release. I’ll be sure to update this post when I have more details.

Regarding upsert behavior, I’ll barrow from my response on thread 1072:

If you want upsert you must specify the unique key. You can alway use the actual entity id, in that case, you would not need to use any identity to perform an update. If you do want to identify an entity by a unique key, you must indeed specify that unique key (not its constituents). This is clean and unambiguous. We are looking at updating our docs to make this clear.

3 Likes
5

We’re trying to use unique composite-tuples (with ref attributes) to automatically generate unique identity attributes for multiple entities upserted in a single transaction.

If a unique attribute is a composite-tuple, normal Datomic upsert behavior doesn’t work unless the value of the composite tuple is provided directly (unique constraint violation on the tuple on all but the first upsert).

As an aside, this doesn’t quite match the docs here

“Composite attributes are entirely managed by Datomic–you never assert or retract them yourself.”

For cases where the unique composite-tuple’s attributes are scalars, this is workable for the use case I’m currently testing because at transaction time, those values are known. I expect this won’t always be true in other cases.

However, if the unique composite-tuple contains attributes that are refs, their entity-ids are not necessarily known if they upserted in the same transaction. Normally we’d use temp-ids in transactions to solve this kind of thing, but entity look-ups fail if temp-ids are used for refs in unique composite-tuples. It seems that specifying the Entity IDs for the ref attributes does work.

If the unique constraint on the composite-tuple is removed, the use of temp-ids works as expected for the composite-tuple’s attributes, but of course a unique entity is not resolved, a new entity is created, thus not achieving the desired upsert behavior.

To work around this, we are maintaining our own entity-id schema, along with corresponding code to generate entity-ids as required; something we’d ideally leave to Datomic if possible.

Are there other solutions that we are missing here?

Could the fact that we can’t use temp-ids when asserting unique composite-tuples directly be considered a bug? Or perhaps that it’s required to assert unique composite-tuples directly at all?

Perhaps there are other ways to achieve the same thing?

Any suggestions would be much appreciated. :slightly_smiling_face:

4 Likes
6

The recommendations for composite tuples are not clear. The documentation states:

Composite attributes are entirely managed by Datomic–you never assert or retract them yourself. Whenever you assert or retract any attribute that is part of a composite, Datomic will automatically populate the composite value.

This seems reasonable since Datomic has the necessary information to construct the tuple itself and duplicating the information in both the constituent attrs and the composite tuple attribute is not clean. Yet upserts do not work when following this recommendation.

Can the documentation be updated -especially if the “I want upsert” recommendation differs from the baseline?

If updating the documentation quickly is an unreasonable request, can someone from Datomic unambiguously address the question here: does upsert of composite tuples generally (including ref attrs!) require contravening the documentation?

2 Likes