We are using Datomic to store Time series data. We get the data hourly.
We have around ~40 billion datoms
When we query ,we try to load the data for last ~5 days.We have defined indexes for data and we use date range based queries.
At times , some query take up to ~1 min.Problem is its very inconsistent
Sadly, We realized Datomic is not optimized for such huge amount of data.
I read this article that give some information about how the data that is stored in Peer cache
I had couple of Queries:
- Is there any way we can identify the levels we need to go to query data.Subsequently , number of trips to storage?
- Is there any direction where we can work on to optimize it?
- why there is inconsistency in time?Sometimes it s fine sometime just take too much time?
Any help will be appreciated.