Troubles with upsert on composite tuples

[brandonbloom]

I’m trying to understand Datomic’s upsert behavior with respect to composite tuple typed attributes when one of the tuple elements is a ref. I’ve created a minimal reproduction of two unexpected behaviors:

  (def schema
    ;; Payload string for debugging.
    [{:db/ident :string
      :db/valueType :db.type/string
      :db/cardinality :db.cardinality/one}
     ;; Top-level of hierarchy, uniquely named items.
     {:db/ident :parent/name
      :db/valueType :db.type/keyword
      :db/cardinality :db.cardinality/one
      :db/unique :db.unique/identity}
     ;; Bottom-level of hierarchy.
     {:db/ident :child/parent
      :db/valueType :db.type/ref
      :db/cardinality :db.cardinality/one}
     {:db/ident :child/name
      :db/valueType :db.type/keyword
      :db/cardinality :db.cardinality/one}
     {:db/ident :child/parent+name
      :db/valueType :db.type/tuple
      :db/tupleAttrs [:child/parent :child/name] ; Uniquely named within parent.
      :db/cardinality :db.cardinality/one
      :db/unique :db.unique/identity}])

  (d/transact conn {:tx-data schema})

  (d/transact conn {:tx-data [{:parent/name :a :string "inserted a"}]})
  (d/transact conn {:tx-data [{:parent/name :b :string "inserted b"}]})
  (d/q '[:find ?parent-name ?string
         :where [?parent :parent/name ?parent-name]
                [?parent :string ?string]]
       (d/db conn))

  (d/transact conn {:tx-data [{:parent/name :a :string "upserted a"}]})
  (d/q '[:find ?parent-name ?string
         :where [?parent :parent/name ?parent-name]
                [?parent :string ?string]]
       (d/db conn))

  (d/transact conn {:tx-data [{:child/parent [:parent/name :a]
                               :child/name :x
                               :string "inserted x"}]})
  (d/q '[:find ?parent-name ?parent-string ?child-name ?child-string
         :where [?parent :parent/name ?parent-name]
                [?parent :string ?parent-string]
                [?child :child/parent ?parent]
                [?child :child/name ?child-name]
                [?child :string ?child-string]]
       (d/db conn))

  ;; ERROR: Unique conflict! But this is expected.
  (d/transact conn {:tx-data [{:child/parent [:parent/name :a]
                               :child/name :x
                               :string "inserted x"}]})

  ;; This doesn't report any errors...
  (d/transact conn {:tx-data [{:db/id "a"
                               :parent/name :a}
                              {:child/parent+name ["a" :y]
                               :string "inserted y"}]})
  ;; But "inserted y" is nowhere to be found!
  (d/q '[:find ?parent-name ?parent-string ?child-name ?child-string
         :where [?parent :parent/name ?parent-name]
                [?parent :string ?parent-string]
                [?child :child/parent ?parent]
                [?child :child/name ?child-name]
                [?child :string ?child-string]]
       (d/db conn))

  ;; ERROR: Invalid tuple value! But would be nice if it worked.
  (d/transact conn {:tx-data [{:child/parent+name [[:parent/name :a] :z]
                               :string "inserted z"}]})

To summarize, the two behaviors I’m puzzled by are:

1. Lookup refs within a tuple cause transact to fail with “Invalid tuple value”.
2. Upserting a tuple with a ref attr seems to silently fail, discarding data. This occurs with both tempid strings and long db ids.

My expectation was that both of these operations would succeed with the following interpretations:

1. Lookup refs would be resolved to long db ids, leaning to #2.
2. Composite tuples would be expanded in to their constitute datums, allowing an upsert on a multi-attribute key.

Some things that lead me to believe that this should work:

• I’m running datomic-pro 0.9.5981 locally & the changelog for that version says “Fix: resolve tempids for reference attributes inside tuples.”
• This comment: :db.unique/identity does not work for tuple attributes

Have I made some error in implementation or understanding?

Is there some other way to upsert on a multi-attribute key? Is it expected to work with ref attributes?

[nikolayandr]

Hi, I had the same issue, even more I asked something similar and the answer was: the list of supported types to be used as value of tuple:

:db.type/bigdec :db.type/bigint :db.type/boolean :db.type/double
:db.type/instant :db.type/keyword :db.type/long :db.type/string
:db.type/symbol :db.type/uri :db.type/uuid

As you see :db.type/ref is not among of this list, but yeah, anyway datomic allows to use db.type/ref, looks like it is automatically resolved to :db.type/long

[jaret]

Hi @brandonbloom,

Thanks for the example. First you’ll want to upgrade to the latest. I am confirming with the dev team, but I believe there was another issue with ref resolution that was only recently fixed. Change Log | Datomic. I might have missed documenting it in our release notes and once I’ve clarified I will update the doc.

Re: your observation:

Composite tuples would be expanded in to their constitute datums, allowing an upsert on a multi-attribute key

From the docs here: Schema Data Reference | Datomic

“Composite attributes are entirely managed by Datomic–you never assert or retract them yourself. Whenever you assert or retract any attribute that is part of a composite, Datomic will automatically populate the composite value.”

Reviewing your example gist, it looks like you are asserting a value for the composite attribute. As the docs above indicate, you should not do that. Instead, assert the values for the two individual elements of the composite and Datomic will make the composite for you.

@nikolayandr, Thank you for pointing out this discrepancy in the docs. But we do support :db.type/ref and I’ll add it to that page shortly.

[brandonbloom]

@jaret

I am confirming with the dev team, but I believe there was another issue with ref resolution that was only recently fixed.

I’ve upgraded to the latest and the behavior I’m seeing is the same as described in my original comment.

@jaret

Reviewing your example gist, it looks like you are asserting a value for the composite attribute. As the docs above indicate, you should not do that.

I originally tried to assert the individual elements, but that causes the problem described in this thread. In short, the transaction will fail with a conflict error. To quote you from over in that thread:

@jaret

If you want upsert you must specify the unique key. You can alway use the actual entity id, in that case, you would not need to use any identity to perform an update. If you do want to identify an entity by a unique key, you must indeed specify that unique key (not its constituents). This is clean and unambiguous. We are looking at updating our docs to make this clear.

After reading that, I tried asserting the composite tuple key directly, which has the problems described in my original post.

@jaret

we do support :db.type/ref and I’ll add it to that page shortly.

Glad to hear this. Was concerned for a moment there, since using refs in composite keys is a critical use case for me!

If you’re saying all this stuff works: Could you please provide a repl log demonstrating how to upsert via a tupleAttrs based multi-attribute unique/identity key?

Thanks!

[onetom]

We keep hitting this limitation with tupleAttrs, so I would also like to see some examples how are we supposed to have a single upsert transactions with tupleAttrs containing :db.type/ref components.

I would expect something like this to be upsertable (transactable twice) without errors, without Unique conflict errors:

[{:ref/x {:db/id "x1" :x/id "x1"}
  :ref/y {:db/id "y1" :y/id "y1"}
  :key   ["x1" "y1"]}]

or like this:

[{:db/id "x1" :x/id "x1"}
            {:db/id "y1" :y/id "y1"}
            {:ref/x "x1"
             :ref/y "y1"
             :key   ["x1" "y1"]}]

where the schema would be:

[{:db/ident       :x/id
  :db/valueType   :db.type/string
  :db/cardinality :db.cardinality/one
  :db/unique      :db.unique/identity}

 {:db/ident       :y/id
  :db/valueType   :db.type/string
  :db/cardinality :db.cardinality/one
  :db/unique      :db.unique/identity}

 {:db/ident       :ref/x
  :db/valueType   :db.type/ref
  :db/cardinality :db.cardinality/one}

 {:db/ident       :ref/y
  :db/valueType   :db.type/ref
  :db/cardinality :db.cardinality/one}

 {:db/ident       :key
  :db/valueType   :db.type/tuple
  :db/tupleAttrs  [:ref/x :ref/y]
  :db/cardinality :db.cardinality/one
  :db/unique      :db.unique/identity}

 {:db/ident       :attr
  :db/valueType   :db.type/long
  :db/cardinality :db.cardinality/one}]

If the entity IDs of x1 & y1 are known, then we can do something like this:

[{:ref/x 123
  :ref/y 456
  :key   [123 456]}]

but it’s impossible to know the entity IDs, unless we already have the referenced entities already transacted in a separate transaction.

[andre]

Another vote to please clarify upsert behaviour for composite tuples.

We do get upsert behaviour when specifying the tuple attribute in the transaction (which goes against the statement in the documentation to never assert tuple attributes). Can we use this without fear that it will stop working in future?

Also, when using a :db.type/ref as part of the tuple, lookup refs are not resolved, meaning we have to query first to resolve all lookup refs. Would be great if lookup refs could work as per non-tuple ref attributes.

[jasonjckn]

Hitting this issue in 2023… +1.