OutOfMemoryError when heavy data query is executed

Datomic Pro
1

We have a query that can trigger a manual Java method for a large number of entities, and because of this, the application has problems with memory allocation, and the executed query ends up with an out-of-memory error.

If I understand correctly, the peer used all the memory to create threads, according to this error: Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached. Is it possible to configure the amount of memory at which the peer should interrupt the execution of the request, or is it possible to configure the maximum number of threads that the service can use to execute the request?

Caused by: java.lang.Exception: processing clause: {:argvars (?ViaxCustomerOrder), :fn #object[datomic.extensions$eval15477$fn__15478 0x4ffabadb "datomic.extensions$eval15477$fn__15478@4ffabadb"], :clause [(io.viax.core.model.query.impl.DynamicFieldValueQueryResolver/getValue $ "wiley" ?ViaxCustomerOrder "bpStatus") ?ViaxCustomerOrder_bpStatus_compare_value_0_VR], :binds [?ViaxCustomerOrder_bpStatus_compare_value_0_VR], :bind-type :scalar, :needs-source true}, message: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at datomic.datalog$eval_clause$fn__5843.invoke(datalog.clj:1428)
	at datomic.datalog$eval_clause.invokeStatic(datalog.clj:1391)
	at datomic.datalog$eval_clause.invoke(datalog.clj:1351)
	at datomic.datalog$eval_rule$fn__5871$fn__5873.invoke(datalog.clj:1478)
	at datomic.measure.query_stats$with_clause_stats.invokeStatic(query_stats.clj:40)
	at datomic.measure.query_stats$with_clause_stats.invoke(query_stats.clj:31)
	at datomic.datalog$eval_rule$fn__5871.invoke(datalog.clj:1478)
	... 154 common frames omitted
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at clojure.core$deref_future.invokeStatic(core.clj:2317)
	at clojure.core$deref.invokeStatic(core.clj:2337)
	at clojure.core$deref.invoke(core.clj:2323)
	at clojure.core$mapv$fn__8553.invoke(core.clj:6993)
	at clojure.lang.PersistentVector.reduce(PersistentVector.java:345)
	at clojure.core$reduce.invokeStatic(core.clj:6899)
	at clojure.core$mapv.invokeStatic(core.clj:6984)
	at clojure.core$mapv.invoke(core.clj:6984)
	at datomic.common$pooled_mapv.invokeStatic(common.clj:700)
	at datomic.common$pooled_mapv.invoke(common.clj:695)
	at datomic.datalog$qmapv.invokeStatic(datalog.clj:52)
	at datomic.datalog$qmapv.invoke(datalog.clj:47)
	at datomic.datalog.FnRel.join_project_with(datalog.clj:667)
	at datomic.datalog$join_project_coll.invokeStatic(datalog.clj:130)
	at datomic.datalog$join_project_coll.invoke(datalog.clj:128)
	at datomic.datalog$fn__5306.invokeStatic(datalog.clj:233)
	at datomic.datalog$fn__5306.invoke(datalog.clj:229)
	at datomic.datalog$fn__5216$G__5209__5231.invoke(datalog.clj:65)
	at datomic.datalog$eval_clause$fn__5843.invoke(datalog.clj:1401)
	... 160 common frames omitted
Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.lang.Thread.start0(Native Method)
2

@nikolayandr can you confirm that the peer application ran out of memory by reviewing metrics of memory on the peer? Can you share the query you believe is causing this issue and the method you are calling along with an estimate for the number of entities this is called for? What are your peer memory settings?

3

@jaret
I can provide only the query now:

[:find 
?ViaxSubscriptionContract 
:in $ ?ViaxSubscriptionContract_bpStatus_value_4 [?ViaxSubscriptionContract_value_5 ...] 
:where 

;; Produces ~10000 elements
	[?ViaxSubscriptionContract :Model/typeName ?ViaxSubscriptionContract_value_5]

;; io.viax.core.model.query.impl.DynamicFieldValueQueryResolver/getValue executes one http call for each entity
	[(io.viax.core.model.query.impl.DynamicFieldValueQueryResolver/getValue $ ?ViaxSubscriptionContract “bpStatus”) ?ViaxSubscriptionContract_bpStatus_compare_value_3]
	
	[(= ?ViaxSubscriptionContract_bpStatus_compare_value_3 ?ViaxSubscriptionContract_bpStatus_value_4)]
]

Unfortunately, I can’t provide the metrics now, but in general, the application becomes unreachable once this error happens so looks like resources are exhausted and the application can’t process the future requests

4

Unfortunately, I can’t provide the metrics now, but in general, the application becomes unreachable once this error happens so looks like resources are exhausted and the application can’t process the future requests

That’s consistent with OOM behavior. Would still be good to capture the metrics so you are sure that is the problem.

@nikolayandr and what is the heap you are giving your application?

Have you tried adding more memory? Alternatively, is there a way to run a smaller query result by splitting the query up?