Annotation using "datomic.tx" issue

Hi, I am a newbie and a bit confused about “datomic.tx”.

For instance, I am transacting:

[[:db/add "datomic.tx" :user/id #uuid "19fb47da-3246-4cfe-924e-de0388e75352"]
[:db/retract 101155069759981 :demo.sales1.product/list-price 112000.00]
[:db/add 101155069759981 :demo.sales1.product/list-price 111000.00]]

and the result is:

{:tx-data [#datom[13194139540131 50 #inst “2023-04-10T09:18:41.255-00:00” 13194139540131 true]], :tempids {“datomic.tx” 87960930224942}}

I would expect to have:

{:tx-data [
#datom[13194139540131 50 #inst “2023-04-10T09:18:41.255-00:00” 13194139540131 true]
#datom[13194139540131 299 #uuid "19fb47da-3246-4cfe-924e-de0388e75352" 13194139540131 true]
], :tempids {“datomic.tx” 13194139540131}}

Am I right?

1 Like

Baffling. Your code looks correct to my eyes.

FWIW, the transaction entity can also be represented as an entity (in “map form”) within the transaction data. For example:

[{:db/id “datomic.tx”
:user/id #uuid “19fb47da-3246-4cfe-924e-de0388e75352”}
[datom 1]
[datom 2]]

But your syntax is valid and should work. This is not a new feature of Datomic, nor a feature limited to Cloud, On-Prem or dev-local/not-dev-local. I can’t explain why you are not seeing the datoms from the transaction entity itself -could you perhaps provide a more complete REPL log of how this is being performed?

Hi, I did try the map syntax as well. Same result. A few more details:

:deps {;; datomic
        com.cognitect/anomalies {:mvn/version "0.1.12"}
        com.datomic/client-cloud {:mvn/version "0.8.113"}
        com.datomic/ion {:mvn/version "0.9.50"}

clj꞉transact꞉> 
[[:db/add "1572" :taia-apps.support.support/chat-id 101155069759056]
 [:db/add "1572" :taia-apps.support.support/problem "ticket 201"]
 [:db/add
  "datomic.tx"
  :user/id
  #uuid "4876ec17-67a3-442f-8309-c0c75611295d"]]

=>

{:tx-data
 [#datom[13194139540162 50 #inst "2023-04-13T06:47:38.831-00:00" 13194139540162 true]
  #datom[87960930228930 267 101155069759056 13194139540162 true]
  #datom[87960930228930 268 "ticket 201" 13194139540162 true]],
 :tempids {"1572" 87960930228930, "datomic.tx" 83562883713271}}

the code:

(defn transact
  ([conn transaction user-id]
     (let [tx (vec (conj (:datoms transaction)
                           [:db/add "datomic.tx" :user/id user-id]))
             _ (clojure.pprint/pprint tx)
             result  (d/transact conn {:tx-data tx})
             _ (clojure.pprint/pprint (select-keys result [:tx-data :tempids]))]
         result)

and in the REPL:

(transact
   (get-connection)
   {:datoms [[:db/add "1572" :taia-apps.support.support/chat-id 101155069759056]
             [:db/add "1572" :taia-apps.support.support/problem "ticket 201"]]}
   #uuid "4876ec17-67a3-442f-8309-c0c75611295d")

I just made a complete example, while trying to reproduce this issue.

(ns xxx
    (:require [datomic.client.api :as d]))

(comment
  (let [client (d/client {:server-type :dev-local
                          :storage-dir :mem
                          :system      "danbunea"})
        db-ref {:db-name "demo"}
        _ (d/create-database client db-ref)
        conn (d/connect client db-ref)]
    (letfn [(tx! [tx-data] (d/transact conn {:tx-data tx-data}))]
      (tx! [{:db/ident       :demo.sales1.product/list-price
             :db/valueType   :db.type/double
             :db/cardinality :db.cardinality/one}
            {:db/ident       :user/id
             :db/valueType   :db.type/uuid
             :db/cardinality :db.cardinality/one}])
      (let [{{product-eid "product"} :tempids}
            (tx! [{:db/id                          "product"
                   :demo.sales1.product/list-price 112000.00}])]
        (tx! [[:db/add "datomic.tx" :user/id #uuid "19fb47da-3246-4cfe-924e-de0388e75352"]
              [:db/retract product-eid :demo.sales1.product/list-price 112000.00]
              [:db/add product-eid :demo.sales1.product/list-price 111000.00]]))))
  )

I couldn’t reproduce it, BUT I saw a different anomaly:

=>
{:db-before #datomic.core.db.Db{:id "46701faa-58ce-472d-813f-36cbd187916f",
                                :basisT 7,
                                :indexBasisT 0,
                                :index-root-id nil,
                                :asOfT nil,
                                :sinceT nil,
                                :raw nil},
 :db-after #datomic.core.db.Db{:id "46701faa-58ce-472d-813f-36cbd187916f",
                               :basisT 8,
                               :indexBasisT 0,
                               :index-root-id nil,
                               :asOfT nil,
                               :sinceT nil,
                               :raw nil},
 :tx-data [#datom[13194139533320 50 #inst"2023-04-13T21:38:45.737-00:00" 13194139533320 true]
           #datom[13194139533320 74 #uuid"19fb47da-3246-4cfe-924e-de0388e75352" 13194139533320 true]
           #datom[101155069755467 73 112000.0 13194139533320 false]
           #datom[101155069755467 73 111000.0 13194139533320 true]
           #datom[101155069755467 73 112000.0 13194139533320 false]],
 :tempids {"datomic.tx" 13194139533320}}

The retraction appears twice in the :tx-data of the d/transact’s return value.
If I remove the explicit retraction from the input tx data, then the output tx data only contains a single retraction, as expected.

Hi, I did write an answer yesterday, but it got caught by a bot and needs approval, so I am rewriting it.

I am using:

 :deps {;; datomic
        com.cognitect/anomalies {:mvn/version "0.1.12"}
        com.datomic/client-cloud {:mvn/version "0.8.113"}
        com.datomic/ion {:mvn/version "0.9.50"}

but it also happens on:

 :deps {;; datomic
        com.cognitect/anomalies {:mvn/version "0.1.12"}
        com.datomic/client-cloud {:mvn/version "1.0.122"}
        com.datomic/ion {:mvn/version "1.0.62"}

Transacting:

clj꞉꞉> 
[[:db/add "1572" :taia-apps.support.support/chat-id 101155069759056]
 [:db/add "1572" :taia-apps.support.support/problem "ticket 202"]
 [:db/add
  "datomic.tx"
  :user/id
  #uuid "4876ec17-67a3-442f-8309-c0c75611295d"]]

results in:

{:tx-data
 [#datom[13194139540165 50 #inst "2023-04-13T07:12:51.957-00:00" 13194139540165 true]
  #datom[101155069762245 267 101155069759056 13194139540165 true]
  #datom[101155069762245 268 "ticket 202" 13194139540165 true]],
 :tempids {"1572" 101155069762245, "datomic.tx" 83562883713271}}
{:datoms
 ([:db/add 101155069762245 :taia-apps.support.support/chat-id 101155069759056]
  [:db/add 101155069762245 :taia-apps.support.support/problem "ticket 202"]),
 :tx {:tx-id 13194139540165, :tx-time #inst "2023-04-13T07:12:51.957-00:00"}}

Running the code exactly as it is:

{:db-before
 #datomic.core.db.Db{:id "c69b34db-5c11-4232-bb05-540429ac0697", :basisT 7, :indexBasisT 0, :index-root-id nil, :asOfT nil, :sinceT nil, :raw nil},
 :db-after
 #datomic.core.db.Db{:id "c69b34db-5c11-4232-bb05-540429ac0697", :basisT 8, :indexBasisT 0, :index-root-id nil, :asOfT nil, :sinceT nil, :raw nil},
 :tx-data
 [#datom[13194139533320 50 #inst "2023-04-14T06:20:30.939-00:00" 13194139533320 true] 
#datom[13194139533320 74 #uuid "19fb47da-3246-4cfe-924e-de0388e75352" 13194139533320 true] 
#datom[101155069755467 73 112000.0 13194139533320 false] #datom[101155069755467 73 111000.0 13194139533320 true] #datom[101155069755467 73 112000.0 13194139533320 false]],
 :tempids {"datomic.tx" 13194139533320}}

Now when using my own production db in the cloud:

    (letfn [(tx! [tx-data] (d/transact (get-connection) {:tx-data tx-data}))] 
        (tx! [[:db/add "datomic.tx" :user/id #uuid "19fb47da-3246-4cfe-924e-de0388e75352"]
              [:db/add "product-eid" :demo.sales1.product/list-price 111000.00]]))

results in:

:db-before
 {:database-id "50020bd5-c6f3-4317-996e-7ea8f2b2b658", :db-name "taia", :t 6865, :next-t 6866, :type :datomic.client/db},
 :db-after
 {:database-id "50020bd5-c6f3-4317-996e-7ea8f2b2b658", :db-name "taia", :t 6866, :next-t 6867, :type :datomic.client/db},
 :tempids {"product-eid" 101155069762258, "datomic.tx" 87960930224942},
 :tx-data
 [#datom[13194139540178 50 #inst "2023-04-14T06:28:49.804-00:00" 13194139540178 true]
  #datom[101155069762258 600 111000.0 13194139540178 true]]}

So, I really don’t know what to say. Except thank you (both) for trying to help.

1 Like

Hello @danbunea !
in your code below, the datomic.tx entity-id 13194139533320 match the id inside tx-data:

{:db-before
 #datomic.core.db.Db{:id "c69b34db-5c11-4232-bb05-540429ac0697", :basisT 7, :indexBasisT 0, :index-root-id nil, :asOfT nil, :sinceT nil, :raw nil},
 :db-after
 #datomic.core.db.Db{:id "c69b34db-5c11-4232-bb05-540429ac0697", :basisT 8, :indexBasisT 0, :index-root-id nil, :asOfT nil, :sinceT nil, :raw nil},
 :tx-data
 [#datom[13194139533320 50 #inst "2023-04-14T06:20:30.939-00:00" 13194139533320 true] 
#datom[13194139533320 74 #uuid "19fb47da-3246-4cfe-924e-de0388e75352" 13194139533320 true] 
#datom[101155069755467 73 112000.0 13194139533320 false] #datom[101155069755467 73 111000.0 13194139533320 true] #datom[101155069755467 73 112000.0 13194139533320 false]],
 :tempids {"datomic.tx" 13194139533320}}

but then when you run on your production db, the datomic.tx entity-id 87960930224942 doesn’t seems to match the id inside tx-data 13194139540178 :

:db-before
 {:database-id "50020bd5-c6f3-4317-996e-7ea8f2b2b658", :db-name "taia", :t 6865, :next-t 6866, :type :datomic.client/db},
 :db-after
 {:database-id "50020bd5-c6f3-4317-996e-7ea8f2b2b658", :db-name "taia", :t 6866, :next-t 6867, :type :datomic.client/db},
 :tempids {"product-eid" 101155069762258, "datomic.tx" 87960930224942},
 :tx-data
 [#datom[13194139540178 50 #inst "2023-04-14T06:28:49.804-00:00" 13194139540178 true]
  #datom[101155069762258 600 111000.0 13194139540178 true]]}

Are you able to query for that user id that you are attempted to assert for that transaction entity ?

Also, what do you find when you query for the entity-id 87960930224942 and for the transaction entity-id 13194139540178 ?

Ok, I seem to be closer to understanding it. First what you asked for:

(d/pull db '[*] 87960930224942)

{:db/id 87960930224942, :user/id #uuid "19fb47da-3246-4cfe-924e-de0388e75352"}

(d/pull db '[*] 13194139540178)

#:db{:id 13194139540178, :txInstant #inst "2023-04-14T06:28:49.804-00:00"}

also

(->> (d/tx-range conn {:start 13194139540178 :end 13194139540179})
       (map :data)
       )

([#datom[13194139540178 50 #inst "2023-04-14T06:28:49.804-00:00" 13194139540178 true]
  #datom[101155069762258 600 111000.0 13194139540178 true]])

Then:

(d/pull db '[*] [:db/ident :user/id])

#:db{:id 88,
     :ident :user/id,
     :valueType #:db{:id 54, :ident :db.type/uuid},
     :cardinality #:db{:id 35, :ident :db.cardinality/one},
     :unique #:db{:id 38, :ident :db.unique/identity}}

and

(d/pull db '[*] [:user/id #uuid "19fb47da-3246-4cfe-924e-de0388e75352"])
{:db/id 87960930224942, :user/id #uuid "19fb47da-3246-4cfe-924e-de0388e75352"}

For each user I have an entity, in which I only keep its uuid (generated by AWS Cognito).
I am trying to reuse the :user/id property to annotate the transaction :thinking: so it seems this is where it gets confused trying the

[:db/add "datomic.tx" :user/id #uuid "19fb47da-3246-4cfe-924e-de0388e75352"]

which I have

[87960930224942, :user/id #uuid "19fb47da-3246-4cfe-924e-de0388e75352"]

in the db, but it shouldn’t…

Ok, managed to reproduce it:

  (let [client (d/client {:server-type :dev-local
                          :storage-dir :mem
                          :system      "danbunea"})
        db-ref {:db-name "demo2"}
        _ (d/create-database client db-ref)
        conn (d/connect client db-ref)]
    (letfn [(tx! [tx-data] (d/transact conn {:tx-data tx-data}))]
      (tx! [{:db/ident       :demo.sales1.product/list-price
             :db/valueType   :db.type/double
             :db/cardinality :db.cardinality/one}
            {:db/ident       :user/id
             :db/valueType   :db.type/uuid
             :db/cardinality :db.cardinality/one
             :db/unique :db.unique/identity}])
      (let [{{product-eid "product"} :tempids}
            (tx! [{:db/id                          "product"
                   :demo.sales1.product/list-price 112000.00}
                  {:db/id "user"
                   :user/id #uuid "19fb47da-3246-4cfe-924e-de0388e75352"}])]
        (tx! [[:db/add "datomic.tx" :user/id #uuid "19fb47da-3246-4cfe-924e-de0388e75352"]
              [:db/retract product-eid :demo.sales1.product/list-price 112000.00]
              [:db/add product-eid :demo.sales1.product/list-price 111000.00]]))))
;; =>
  {:db-before
   #datomic.core.db.Db{:id "7a0cd0d6-6d38-4fd6-aa33-2e5d6be6c06f", :basisT 7, :indexBasisT 0, :index-root-id nil, :asOfT nil, :sinceT nil, :raw nil},
   :db-after
   #datomic.core.db.Db{:id "7a0cd0d6-6d38-4fd6-aa33-2e5d6be6c06f", :basisT 8, :indexBasisT 0, :index-root-id nil, :asOfT nil, :sinceT nil, :raw nil},
   :tx-data
   [#datom[13194139533320 50 #inst "2023-04-16T08:30:08.106-00:00" 13194139533320 true] 
    #datom[101155069755467 73 112000.0 13194139533320 false] 
    #datom[101155069755467 73 111000.0 13194139533320 true] 
    #datom[101155069755467 73 112000.0 13194139533320 false]],
   :tempids {"datomic.tx" 101155069755468}}

It’s the
:db/unique :db.unique/identity
or the user id.

If that user already exists in the db, it won’t be added to the transaction. Maybe some warning should be shown here, but at least now I know what it is.

I’ll make another attribute to annotate in the tx.

that’s a funny situation u just outlined! :slight_smile:

indeed the solution is to create another attribute, since a transaction entity is NOT a user itself, so it shouldn’t have :user/* attributes. It just references a user.

I just recently pondered what should be our naming convention for attributes, which are used to describe Datomic transaction entities and I came up with :datomic.tx/*, which is not prohibited, reasonably concise and has 2-segment namespace, which makes it very unlikely to collide with other names.

Following that idea, your attribute could be called something like

  • :datomic.tx/by
  • :datomic.tx/origin
  • :datomic.tx/source

Either way, you can also use such an attribute to point to some entity, which represents some system, not necessarily just a human user.

Btw, I strongly discourage everyone to just talk about users (or accounts), without any further qualifications. Most system sooner or later will need to deal with more types of user or accounts and the ambiguity quickly becomes a serious communication hinderance.

For example, I work for https://gini.co, so we have :gini/users, to whom we assign some :gini.user/id UUID attribute, which we generate, but in your case it could be the Cognito user ID (u might even have such an attribute, :cognito.user/id or even :aws.cognito.user/id).

Then these gini users sign up to our system with either their Google user accounts or Xero user accounts (notice how the word account entered the picture!). Those systems have their own user IDs, which we store as :google.user/id & :xero.user/id.

Then these gini users pay us using Stripe, so they will have a :stripe.customer/id.
However, we are planning to allow our users - the gini users - to share their own Stripe data with our system. For that, they will have to use OAuth authorization, after signing in with their Stripe user credentials to Stripe. Such an authorization is then implicitly tied to their Stripe user account, which they used as an :oauth.client/user (or just :oauth/user), which would point to a Stripe user entity, which would have a :stripe.user/id.

Xero is an accounting system, so naturally it manages “accounts”, but in the financial sense!
So while they have user accounts, they have financial accounts too!
We ended up calling those accounting accounts, for the sake of disambiguation, because we also use HubSpot, which has a bunch of other (user) account types:

Marketing departments also calls have a special meaning for accounts; they mean a logical entity, which is comprised of 1 or more legal entities and 1 or more contact persons, called contacts for short. If I would need to model them in Datomic, I’d use :hubspot.marketing.account/id, :hubspot.user/id, :hubspot.contact/id.

If an employee of a company becomes a user of the company’s product, then i would even mix their :gini.user/id & :hubspot.contact/id on the same entity (if they would be in the same Datomic database).

There is adjacent problem with unique tuple attrs (or composite keys, as i prefer to call them).
Since composite key attrs are implied, you can not just use the same constituent attributes across multiple composite keys.

For example, imagine you want to store a copy of some 3rd-party data (let’s say invoices & parties) from a multi-tenant system.

You would need 3 kind of entities:

  1. company (the tenant, from the 3rd-party system’s view)
  2. invoice
  3. party

Invoice and party IDs are not globally unique, but unique only within a tenant, so they would need composites keys, which link them to their corresponding company. So let’s consider the following attrs:

  • :xxx.company/id (str)
  • :xxx/company (ref)
  • :xxx.invoice/id (str)
  • :xxx.invoice/company+id (tuple)
  • :xxx.party/id (str)
  • :xxx.party/company+id (tuple)

If the composite key attrs are defined like this:

  {:db/ident :xxx.invoice/company+id
   :db/tupleAttrs [:xxx/company :xxx.invoice/id]
   ,,,}

  {:db/ident :xxx.party/company+id
   :db/tupleAttrs [:xxx/company :xxx.party/id]
   ,,,}

then transacting a party would automatically contain a
[... :xxx.invoice/company+id [<company eid> nil] ... true]
datom too, besides the expected
[... :xxx.party/company+id [<company eid> "<party-id>"] ... true]
one.

So you would need to create a per-entity attr, which references the company, instead of a generic :xxx/company ref attr.

  • :xxx.invoice/company
  • :xxx.party/company

and define the composite keys like this instead:

  {:db/ident :xxx.invoice/company+id
   :db/tupleAttrs [:xxx.invoice/company :xxx.invoice/id]
   ,,,}

  {:db/ident :xxx.party/company+id
   :db/tupleAttrs [:xxx.party/company :xxx.party/id]
   ,,,}

Now I am using :tx/user-id which corresponds to the :cognito.user/id. But now you got me thinking. We recently had the following issue: a company needs a worker for a certain period and sends several proposals to a few workers. When the first one accepts, the other proposals get automatically cancelled. In the history, we had the cancelled proposals as cancelled by the user who accepted as he was the one triggering the action, which was wrong, as the auto cancel was done by the system.

why don’t you compute the status of the proposal, instead of storing it?

just be more functional and you wouldn’t even need that background process, plus the state transition would also be atomic.

in fact, why don’t u cancel the other proposals already at the time of approval?
just have a transaction function, which receives a reference to the proposal to be accepted, then it would just look up the other proposals and mark them rejected.

but either way, it feels like the data model is a bit off, because it allows representing states, which are not desired and then u need to write extra code to prevent those states.

u can define computed attributes via pull patterns, using the combination of :xform, :as & (reverse) lookup refs. i have the gut feeling that your situation could be solved by such a technique in a lot simpler way.

alternatively, u can have computed attributes via datalog queries too, though you would need to fold them into the map of a pulled entity. (maybe u can use some custom query function to do that, as opposed to a query post-processing step?)

I was under the impression that you are not supposed to use the :tx namespace in your own attribute names, but I don’t know why did I think that, where did I “hear” that.
Both Datomic docs only mention the :db & :db.* namespaces being reserved.

  1. Schema Limits | Datomic
  2. Schema Limits | Datomic