Cache behaviour on production topology


#1

Hi,

I am currently working on an ions based application deployed on a dedicated query group in a production topology. I am a bit confused by the behaviour of the cache. Indeed once a new revision of the ions is deployed, it can take between a few seconds and several hours to see new transactions (even the ones performed by the ions). Once new transactions are available, the problem does not seem to occur until a new deployment. I do not experience such behaviour when I am working locally.

Is there a way to configure the cache in order to have less delay between a deployment and the availability of new transactions ?

Environment:

  • Datomic Cloud revision (storage, compute, query group): 454-8573
  • Datomic ion: 0.9.28
  • Clojure: 1.9.0 and 1.10.0
  • Compute instance type: 2 i3.large
  • Query group instance type: 1 t2.medium
  • Query Group has no preload of database
  • AWS region: eu-central-1

Thanks


#2

Can you be more specific about what your application is doing when it “can’t see” newer transactions?

Are you able to run queries but you do not get the expected results? Do queries run directly on the compute instances (i.e. not query group instances) return different results?

I would also be interested in taking a look at your ion code and/or your CloudWatch logs and metrics over the period where you see this inconsistency. If you would be willing to provide read only access to your CloudWatch logs, can you email me (marshall@cognitect.com) to set that up?

Thanks!


#3

Hi,

The application is reacting to an http request through a web service ion and issue a Datomic query to list specific entities. The query runs well, however, I kind of get stuck in the version of the database on which the first query was performed. Indeed, if transactions are then performed to add or retract entities, I cannot see the new version of the database by issuing again the request to list the entities. This situation can last for minutes to hours and then for some ignored reason the application get access to the latest version of the database and I can see the result of all the transactions. The problem can not be reproduced then until a new deployment.

I deployed the same ion working on the same database on the compute instances and I cannot reproduce the problem. Moreover, if I use the compute instances and query group deployments at the same time I get different results. The compute instances deployment always return results from the freshest version of the database while the query group deployment get stuck on the old version.

I hope this is clear, I struggle a bit to find a proper way to describe the problem.

I will send you a mail for the CloudWatch access.

Thanks


#4

We’ve reproduced this behavior and are working on a fix that should be included in the next release.

Thanks for catching this!