Syncing to datascript

Were trying to sync part of datomic to datacript, and we ran into an issue that datomic id’s produced are too big for datascript.

I believe the best solution likely involves syncing around a uuid rather then the db/id. This fixes the issue with the servers db/id being too long on the client and it generically fixes the issue of uniquely identifying an entity across multiple databases.

One downside is that we have to add a UUID to every entity in the db. And when we transact data to datomic,that we then need to ship to datascript, we would need a way to get that uuid (as it wouldn’t be in the tx-data log sense it wasn’t touched). However, that could just be done with a lookup, probably not too slow.

Alternatively, if we don’t want to add any facts to the Datomic, we could sync over an in memory unique.value client side attribute called :server/id, that is the :db/id from the server.

One common issue in both approaches is modifying the transaction log to add information (the uuid or server/id). The only idea i have is to do a none side effecting transaction using d/db-after and get the tx-data form that which always seems to be a log as opposed to the nested maps that can be submitted to transact! I just assume it will be hard to properly parse nested map tx’s so using the log will be far easier.

The code to change the data script client tx-data log to add this server/id currently taking this shape:

   (fn [tx-data idx [too-long-server-db-id & rest-of-datom]]
     (let [temp-id (str idx)]
       (conj tx-data
             (vec (conj rest-of-datom temp-id :db/add)) ;; we replace the server id which is too long with a temporary id based on the idx of the log
             [:db/add temp-id :server/id (str too-long-server-db-id)])))

   (vec (take 1 tx-data)))
  ;; => #{[:db/add "0" :server/id "316659348816869"]
  ;;      [:db/add
  ;;       "0"
  ;;       :internal-team/id
  ;;       #uuid "5caf6e45-f54d-4c0e-9658-33f63c069569"
  ;;       536870913
  ;;       true]}

If you have any thoughts, let me know :slightly_smiling_face: (yes, I see that fact has a UUID, but they aren’t on everything!).

  • Here is a slack discussion on the topic that some might find useful
  • datasync does this, maybe i could pull inspiration from there, i wonder if it predates unique.value in datascript though, or at least why they wouldn’t use that (i don’t see it mentioned.

I ended up forking datascript and bumping the emax number, as it was just a perf hit.