Stale data with dev storage protocol?

Datomic Pro Troubleshooting
1

I’m using on-prem and the dev storage protocol running locally for development, with the client api.

I’m rewriting a traditional (no-js) web application to use Datomic and Ring. I’m using a POST request which issues a transaction to update (“accumulate”) a datomic entity, which then immediately redirects to a GET route. That route then immediately issues a query for the data that’s just been written. (shared nothing pattern).

Very frequently the query in the GET request does not reflect the data which was written immediately previously. If I hit reload on the page again I see the changed state. I’ve printed to console at various points and am sure this is happening, browser cache is off etc.

I was under the impression that if I issue d/transact then when that command completes I can be confident that the database has “got it”. i.e., On the very next line of code, I can get a new database value and query against it and this value will reflect what I just transacted on the line before.

Is my assumption correct? Is this something to do with the dev storage protocol? The proportion of times I get the stale data has risen a lot when I try it again this morning, perhaps my jvms are a lot more sleepy at 8am Sunday morning… :slight_smile:

Thinking about the architecture of datomic with the tx data having to reach the peer-server instantly, I can see there’s more going on here than in a system where reads & writes go through the same point.

Question though, how can I ensure I get fresh data ?!

Thanks,
Mike

2

You can use datomic.api/sync to block the peer to a known time-basis (e.g. a time-basis returned by datomic.api/transact but perhaps from a different process). Are you at scale? In a dev configuration with one peer and one transactor it is a bit odd that you are seeing this. In a multi-peer system I believe you can expect peer-time to be within about 10ms of transactor-time.

1 Like
3

Thanks Dustin! I got the basis t out of the transaction report, and passed it on to the next route via a query string, then in place of the usual (d/db conn) did (sync conn basis-t) to get the synced database value instead.

I also print the basis used in each read & write to console and can clearly see the read falling behind the immediately prior write basis.

However, while I can get an example call to sync working in the repl, in the browser I always get error “Datomic Client Exception async.clj datomic.client.api.async/ares”. I don’t know what to do next at the moment.

I tried an alternate quick fix by just delaying the redirect with Thread/sleep. Even upping that to 1000ms wasn’t enough to reliably see fresh data on one tx I was working on.

Update later in the day:
I’ve found that, without using sync (having failed to set that up thus far across an http redirect), whether or not the need to sync occurs is highly dependent on which attribute I’m updating. Specifically, when I toggle in my case a “series” attribute between “published” and “draft”, vs. an identical “lesson” attribute between “published” and “draft”, in every case out of hundreds the lesson attribute doesn’t give any sync/race problem and this is with no extra code to sync the basis added.

But with another attribute, about 20% of the time there’s a race. A third attribute, 100% of the time!

Just what is getting ahead of what here is a bit mind boggling to contemplate, and I don’t propose to speculate, however – I do recall in one of the “Day of Datomic” presentations Stu mentioning that if someone came to them with a problem and they asked what storage they were using and the answer was “dev”, they’d say, we’re not answering, use real storage.

Question - is non-dev storage needed for much beyond getting started & demos? Is that recommended here?

Partial solution - I used the “carefully turn each visible screw to see what happens” method and found it was my use of :next-t of :db-after which was causing client exception. Thus if client had the power of speech the error might have been “you gave me a minimum basis that I haven’t reached yet, and i got tired waiting”. Of course it’s entirely my oversight. I just grabbed a t without much thinking about it the first time around. The lack of error messages isn’t that bad here dare I say it :slight_smile: - just stepping back and reasoning it out, it’s clear it’s the :t of the :db-after that’s what’s needed as the minimum basis, not the :next-t of the :db-after. That’s one further into the future, which in this case isn’t reached, hence the (allbeit opaque) exception.