Troubles with upsert on composite tuples

I’m trying to understand Datomic’s upsert behavior with respect to composite tuple typed attributes when one of the tuple elements is a ref. I’ve created a minimal reproduction of two unexpected behaviors:

  (def schema
    ;; Payload string for debugging.
    [{:db/ident :string
      :db/valueType :db.type/string
      :db/cardinality :db.cardinality/one}
     ;; Top-level of hierarchy, uniquely named items.
     {:db/ident :parent/name
      :db/valueType :db.type/keyword
      :db/cardinality :db.cardinality/one
      :db/unique :db.unique/identity}
     ;; Bottom-level of hierarchy.
     {:db/ident :child/parent
      :db/valueType :db.type/ref
      :db/cardinality :db.cardinality/one}
     {:db/ident :child/name
      :db/valueType :db.type/keyword
      :db/cardinality :db.cardinality/one}
     {:db/ident :child/parent+name
      :db/valueType :db.type/tuple
      :db/tupleAttrs [:child/parent :child/name] ; Uniquely named within parent.
      :db/cardinality :db.cardinality/one
      :db/unique :db.unique/identity}])

  (d/transact conn {:tx-data schema})

  (d/transact conn {:tx-data [{:parent/name :a :string "inserted a"}]})
  (d/transact conn {:tx-data [{:parent/name :b :string "inserted b"}]})
  (d/q '[:find ?parent-name ?string
         :where [?parent :parent/name ?parent-name]
                [?parent :string ?string]]
       (d/db conn))

  (d/transact conn {:tx-data [{:parent/name :a :string "upserted a"}]})
  (d/q '[:find ?parent-name ?string
         :where [?parent :parent/name ?parent-name]
                [?parent :string ?string]]
       (d/db conn))

  (d/transact conn {:tx-data [{:child/parent [:parent/name :a]
                               :child/name :x
                               :string "inserted x"}]})
  (d/q '[:find ?parent-name ?parent-string ?child-name ?child-string
         :where [?parent :parent/name ?parent-name]
                [?parent :string ?parent-string]
                [?child :child/parent ?parent]
                [?child :child/name ?child-name]
                [?child :string ?child-string]]
       (d/db conn))

  ;; ERROR: Unique conflict! But this is expected.
  (d/transact conn {:tx-data [{:child/parent [:parent/name :a]
                               :child/name :x
                               :string "inserted x"}]})

  ;; This doesn't report any errors...
  (d/transact conn {:tx-data [{:db/id "a"
                               :parent/name :a}
                              {:child/parent+name ["a" :y]
                               :string "inserted y"}]})
  ;; But "inserted y" is nowhere to be found!
  (d/q '[:find ?parent-name ?parent-string ?child-name ?child-string
         :where [?parent :parent/name ?parent-name]
                [?parent :string ?parent-string]
                [?child :child/parent ?parent]
                [?child :child/name ?child-name]
                [?child :string ?child-string]]
       (d/db conn))

  ;; ERROR: Invalid tuple value! But would be nice if it worked.
  (d/transact conn {:tx-data [{:child/parent+name [[:parent/name :a] :z]
                               :string "inserted z"}]})

To summarize, the two behaviors I’m puzzled by are:

  1. Lookup refs within a tuple cause transact to fail with “Invalid tuple value”.
  2. Upserting a tuple with a ref attr seems to silently fail, discarding data. This occurs with both tempid strings and long db ids.

My expectation was that both of these operations would succeed with the following interpretations:

  1. Lookup refs would be resolved to long db ids, leaning to #2.
  2. Composite tuples would be expanded in to their constitute datums, allowing an upsert on a multi-attribute key.

Some things that lead me to believe that this should work:

Have I made some error in implementation or understanding?

Is there some other way to upsert on a multi-attribute key? Is it expected to work with ref attributes?

2 Likes

Hi, I had the same issue, even more I asked something similar and the answer was: the list of supported types to be used as value of tuple:

:db.type/bigdec :db.type/bigint :db.type/boolean :db.type/double
:db.type/instant :db.type/keyword :db.type/long :db.type/string
:db.type/symbol :db.type/uri :db.type/uuid

As you see :db.type/ref is not among of this list, but yeah, anyway datomic allows to use db.type/ref, looks like it is automatically resolved to :db.type/long

Hi @brandonbloom,

Thanks for the example. First you’ll want to upgrade to the latest. I am confirming with the dev team, but I believe there was another issue with ref resolution that was only recently fixed. https://docs.datomic.com/on-prem/changes.html#0.9.6024. I might have missed documenting it in our release notes and once I’ve clarified I will update the doc.

Re: your observation:

Composite tuples would be expanded in to their constitute datums, allowing an upsert on a multi-attribute key

From the docs here: https://docs.datomic.com/cloud/schema/schema-reference.html#composite-tuples

“Composite attributes are entirely managed by Datomic–you never assert or retract them yourself. Whenever you assert or retract any attribute that is part of a composite, Datomic will automatically populate the composite value.”

Reviewing your example gist, it looks like you are asserting a value for the composite attribute. As the docs above indicate, you should not do that. Instead, assert the values for the two individual elements of the composite and Datomic will make the composite for you.

@nikolayandr, Thank you for pointing out this discrepancy in the docs. But we do support :db.type/ref and I’ll add it to that page shortly.

I’ve upgraded to the latest and the behavior I’m seeing is the same as described in my original comment.

I originally tried to assert the individual elements, but that causes the problem described in this thread. In short, the transaction will fail with a conflict error. To quote you from over in that thread:

After reading that, I tried asserting the composite tuple key directly, which has the problems described in my original post.

Glad to hear this. Was concerned for a moment there, since using refs in composite keys is a critical use case for me!

If you’re saying all this stuff works: Could you please provide a repl log demonstrating how to upsert via a tupleAttrs based multi-attribute unique/identity key?

Thanks!