Hi guys,
When using the Client API (with Datomic Cloud or locally with a Peer Server), is there any way to use the database functions?
I have 2 use cases:
Currently, I am getting the following exception when trying to add a function built with (datomic.api/function …):
java.lang.RuntimeException: java.lang.Exception: Not supported: class clojure.lang.Delay
Many thanks,
Mykola
Mykola,
With Datomic Cloud, you’ll need to create your functions as Ions (https://docs.datomic.com/cloud/transactions/transaction-functions.html#custom and https://docs.datomic.com/cloud/ions/ions.html) in order to deploy them to your running Cloud instance.
In Datomic On-Prem, you would need to install your database function in a transaction (https://docs.datomic.com/on-prem/database-functions.html). The day-of-datomic examples include some Peer library code demonstrating this (https://github.com/Datomic/day-of-datomic/blob/master/tutorial/data_functions.clj).
I’m currently prototyping with on-prem, peer server & the client lib, and looking to install some transaction functions. To do this, do I have to also run a peer and use the peer library? (have only used client so far) i.e., am I right in guessing that tx functions cannot be installed via client and peer-server alone?
Further background, I can bin/repl in the on-prem install directory into what I believe is the transactor (that’s where I run create-database and delete-database), but am I right in thinking that’s not where we’d install a transaction function - it’s assumed these would be sent in via the peer library only?
The docs do also suggest an alternate method - via the transactor’s classpath (which I note is the only way to install user tx functions in datomic cloud), but I’m not sure how to do that; would need a bit more detailed guidance / exact steps as the docs don’t go into detail on how to do that.
FYI I am thinking of switching to cloud soonish, but still developing with on-prem currently.
Update: there is a lot of good and detailed explanation of how tools.deps works and I could likely figure this out with a lot of effort (I don’t have ten years of production java experience like many clojure devs do). What I haven’t found is a tutorial for installing a classpath transaction function for on-prem in the style of “do this”, “then this”, “type this”, etc.
Hmm… perhaps because of the diversity of build tools there’s “no one way to do this”?
Apologies for my ignorance re. the above - I discovered I had misconceptions re. the whole Clojure compilation story, getting in the way of understanding. Fixed that now (In particular I hadn’t grasped the dynamic compilation clojure is doing from .clj files to bytecode via ASM - my misconception had been that something like javac had to occur for every piece of clojure code, which isn’t true… - makes much more sense actually, I’d wondered how a lisp image could be maintained that way!).
So, lein jar does the trick (or e.g. badigeon with tools.deps), for this AOT compile, then we set the env var DATOMIC_EXT_CLASSPATH as per the docs, & restart the transactor.
So it appears I’ve done all that right, except there’s now another problem…
@Marshall Could you confirm that classpath transaction functions are not supported at all via the client api with on-prem? I see that this may have been possible to read into your reply of Dec '18 above, but I went ahead anyway and installed (as far as I can tell correctly) some class path functions into datomic by adding a jar to the classpath.
However, when I try to use the client api to call them, by first requiring in my tx functions library (noting that classpath transaction functions are invoked with the symbol referring to the function, not via a keyword as database functions are), then issuing like:
[my.ns/my-function 1 2 3],
I get the response:
"Not supported: class my.ns$my-function".
I suspect what may be the case is that, with client, and on-prem, the only way to call a transaction function is of the database function variety (not the classpath variety), and to be able to install those one must use the peer library (which I haven’t yet ventured into, as my getting started experience up to now I have chosen to limit to on-prem and the client api).
If this analysis is correct, I wonder what the most straightforward route forward would be. One option is to introduce the peer library to my on-prem dev setup, else, it looks like I may have to hold off on transaction functions till I migrate wholesale to cloud, as has always looked probable.
Any suggestions would be very welcome.
Thanks, Mike
Yes, that’s correct - you would need to use the peer API to install your transaction function(s), but you can use them from the Client API once they’re installed:
;; In a REPL that has the Peer API in the classpath,
;; create, install, and test the txn function:
(require
'[datomic.api :as d])
(def uri "datomic:dev://localhost:4334/tx-fn-test")
(d/create-database uri)
(def conn (d/connect uri))
(def ensure-composite
(d/function
'{:lang "clojure"
:params [db k1 v1 k2 v2]
:code (if-let [[e t1 t2] (d/q '[:find [?e ?t1 ?t2]
:in $ ?k1 ?v1 ?k2 ?v2
:where
[?e ?k1 ?v1 ?t1]
[?e ?k2 ?v2 ?t2]]
db k1 v1 k2 v2)]
(throw (ex-info (str "Entity already exists " e)
{:e e :t (d/tx->t (max t1 t2))}))
[{:db/id (d/tempid :db.part/user)
k1 v1
k2 v2}])}))
;; install a transaction function
@(d/transact conn [{:db/id #db/id [:db.part/user]
:db/ident :examples/ensure-composite
:db/doc "Create an entity with k1=v1, k2=v2, throwing if such an entity already exists"
:db/fn ensure-composite}])
(def db (d/db conn))
;; test locally
(d/invoke db :examples/ensure-composite db
:db/ident :example/object :db/doc "Example object")
(def tx-data [[:examples/ensure-composite :db/ident :example/object :db/doc "Example object"]])
;; first ensure wins
@(d/transact conn tx-data)
;; second ensure throws exception including t at which entity is known
(try
@(d/transact conn tx-data)
(catch Exception e
(println "Got expected exception " (.getMessage e))
(def t (:t (ex-data (.getCause e))))))
;;; End Peer REPL session ;;;
;;;; From a console run the peer server:
; bin/run -m datomic.peer-server -a k,s -d test,datomic:dev://localhost:4334/tx-fn-test
;;; In a REPL that has the Client API in the classpath,
;;; You can use the installed txn function:
(require '[datomic.client.api :as d])
(def client
(d/client
{:server-type :peer-server
:endpoint "localhost:8998"
:secret "s"
:access-key "k"
:validate-hostnames false}))
(def conn (d/connect client {:db-name "test"}))
(def tx-data [[:examples/ensure-composite :db/ident :example/object2 :db/doc "Example object"]])
;; first ensure wins
(d/transact conn {:tx-data tx-data})
;; second ensure throws exception including t at which entity is known
(try
(d/transact conn {:tx-data tx-data})
(catch Exception e
(println "Got expected exception " (.getMessage e))
(def t (:t (ex-data (.getCause e))))))
Thanks @marshall, I’m up and running with this now. Great!
While limiting my getting started experience with on-prem to the client lib, it hadn’t sunk in what bin/repl in the on-prem install directory was (where I’m accustomed to running create-database and delete-database - but as dev setup says, this is “a Clojure REPL that includes the Datomic Peer library in the classpath”) (perhaps it’s not a “real peer” until we’ve added caching? - not sure on that one).
I’m not sure if the following is a silly question or not, but can I connect to datomic’s bin/repl remotely? The notion here is to set up a run configuration in Intellij/Cursive to enable me to evaluate forms from a source file in my project into that repl without needing to add the peer dependency to my project, as I may not need to otherwise (might be wrong on that). (I have already added it and had to downgrade the client lib version I was using to get it to work alongside the on-prem version I have installed, which is one less than the current version - so solved that conflict, but perhaps this just illustrates why it’s good to avoid dependencies where possible).
There are of course many types of repls and the subject is complex. I don’t see a port being written to the file system when I start bin/repl, and from looking at the shell scripts in there it appears to be a clojure main style of REPL, not an nREPL.
Does this seem correct to you? Thanks so much.
@marshall Sorry I have another follow up.
I’m having trouble using d/invoke (i.e. on the peer api), to test the database function. I was passing it a d/with-db made by the client api, simply because that followed the scheme of my existing tests.
On reflection it looks like mixing these two may not be a good idea.
Passing a with-db made by client to d/invoke:
; as param to the db function:
Execution error (ClassCastException) at datomic.api/pull (api.clj:164).
datomic.client.impl.shared.Db cannot be cast to datomic.Database
; as first param to invoke:
Execution error (ClassCastException) at datomic.db/invoke (db.clj:1889).
datomic.client.impl.shared.Db cannot be cast to datomic.db.IDb
I tried using d/with on peer instead, but the problem there is that a with db originating from peer seems to fail when I send it to a call to d/q implemented with client:
; calling d/q of client with a with db made by peer
Execution error (ExceptionInfo) at datomic.client.api.impl/incorrect (impl.clj:42).
Query args must include a database
Is there a general/conceptual point here to know about, on not mixing database values, speculative or other, between client and peer?
I have a sneaking feeling that in trying to do so I may have been asking the impossible (architecturally, perhaps).
Or would upgrading everything help? - I’m on a slightly older version of client which I bumped down until I upgrade my on-prem to the most recent release.
Update
I’ve just realised I can strip out the need to send a database value from within the transaction function - converting the code which needed that to a transaction function itself was already in the plan. (changing the wheels of the car while it’s moving here…).
That possibly suggests I should be able to use peer with-dbs exclusively for d/invoke.
So, to conclude, question remains whether with dbs from client are impossible to send to d/invoke on peer, if you could confirm? (an architectural explanation of why this is obviously not possible might help… if it’s not possible…).
@marshall maybe information that deploying of functions in peer and client is a point to be mentioned here Clients and Peers | Datomic ?