Dynamic queueing in Datomic

I’m evaluating Datomic Cloud as an alternative to a relational database we’re
using in one of our applications. The current approach is presenting us with
challenges that I think Datomic will solve quite neatly.

At the heart of the application is a dynamic queue: query the database for a
set of items and assign those items to a user. It’s important to prevent
items from being assigned more than once.

Our current approach runs both a query and insert in a DB transaction. This
is easy enough to do in the application code, but to do the same in Datomic
Cloud it looks like I’d have to deploy the code and run the query in a
transaction function. I’m a little nervous about this because the query is
potentially quite complex, and will eventually rely on inputs to control its
behaviour.

Are transaction functions the right tool to use here, or is
there another way? If they are, is there a way to easily test complex transaction
functions without having to deploy on each change?

Thanks,
Luke

Not sure about your first question, but

is there a way to easily test complex transaction functions without having to deploy on each change?

The best way I was able to figure out for doing this is to set up some code to evaluate the transaction functions locally before sending them to the transactor. Pretty janky but it worked for my small app, not sure I’d recommend it for everyone.

(This approach is only for if you want to test your transaction function by actually running transactions, presumably on a dev db. If you’re ok with just unit tests, then you could just test them like any other pure function)

Transaction functions should be pure functions that return transaction data, so REPL testing is definitely appropriate: https://docs.datomic.com/cloud/transactions/transaction-functions.html#testing

1 Like

Is the current approach you use something along the lines of using the transaction to “claim” the particular item?

If so, you do have a couple of options for handling this requirement in Datomic.
As you mentioned, a transaction function is definitely one approach. Can you quantify “potentially quite complex” and “rely on inputs”? If you mean that your query may be differentially parameterized, this could still be fine for a transaction function, but if it’s likely you’ll be needing, for example, remote service calls or something along those lines, you’re right that is a misfit for a transaction function.

I would also consider the possibility of using a compare-and-swap model for “claiming”. Depending on your likelihood of collisions, you may be quite successful with a more “optimistic concurrency” approach of running the query (in an Ion or via the client), and building up your transaction that does a CAS (https://docs.datomic.com/cloud/transactions/transaction-functions.html#db-cas) to “claim” the item. That way if another process attempts to do the same, only one or the other will succeed.

-Marshall

1 Like

Thanks both.

Marshall, yes that’s exactly right. The chances of collisions when claiming an item are quite small. It would be quite easy to add retry logic in the code, so I think CAS could work quite nicely. By complex I mean the queueing behaviour is determined by a set of criteria, so yes you’re right there, too. No requirement to make remote service calls.

Jacob, nice to know that the option for testing transaction functions does exist. Trawling through the docs this past week I’ve definitely felt like guidance on the Datomic developer experience could be improved upon.