# OutOfMemoryError when heavy data query is executed

**URL:** <https://forum.datomic.com/t/outofmemoryerror-when-heavy-data-query-is-executed/2280>\
**Category:** Datomic Pro\
**Created:** [June 30, 2023, 10:51am UTC](https://forum.datomic.com/t/outofmemoryerror-when-heavy-data-query-is-executed/2280 "2023-06-30T10:51:02Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nikolayandr](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/nikolayandr/32/375_2.png) [@nikolayandr](https://forum.datomic.com/u/nikolayandr)\
**Post date:** [June 30, 2023, 10:51am UTC](https://forum.datomic.com/t/outofmemoryerror-when-heavy-data-query-is-executed/2280/1 "2023-06-30T10:51:02Z")

</div>

We have a query that can trigger a manual Java method for a large number of entities, and because of this, the application has problems with memory allocation, and the executed query ends up with an out-of-memory error.

If I understand correctly, the peer used all the memory to create threads, according to this error: **Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached**. Is it possible to configure the amount of memory at which the peer should interrupt the execution of the request, or is it possible to configure the maximum number of threads that the service can use to execute the request?

```auto
Caused by: java.lang.Exception: processing clause: {:argvars (?ViaxCustomerOrder), :fn #object[datomic.extensions$eval15477$fn __15478 0x4ffabadb "datomic.extensions$eval15477$fn__ 15478@4ffabadb"], :clause [(io.viax.core.model.query.impl.DynamicFieldValueQueryResolver/getValue $ "wiley" ?ViaxCustomerOrder "bpStatus") ?ViaxCustomerOrder_bpStatus_compare_value_0_VR], :binds [?ViaxCustomerOrder_bpStatus_compare_value_0_VR], :bind-type :scalar, :needs-source true}, message: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at datomic.datalog$eval_clause$fn__5843.invoke(datalog.clj:1428)
	at datomic.datalog$eval_clause.invokeStatic(datalog.clj:1391)
	at datomic.datalog$eval_clause.invoke(datalog.clj:1351)
	at datomic.datalog$eval_rule$fn __5871$fn__ 5873.invoke(datalog.clj:1478)
	at datomic.measure.query_stats$with_clause_stats.invokeStatic(query_stats.clj:40)
	at datomic.measure.query_stats$with_clause_stats.invoke(query_stats.clj:31)
	at datomic.datalog$eval_rule$fn__5871.invoke(datalog.clj:1478)
	... 154 common frames omitted
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at clojure.core$deref_future.invokeStatic(core.clj:2317)
	at clojure.core$deref.invokeStatic(core.clj:2337)
	at clojure.core$deref.invoke(core.clj:2323)
	at clojure.core$mapv$fn__8553.invoke(core.clj:6993)
	at clojure.lang.PersistentVector.reduce(PersistentVector.java:345)
	at clojure.core$reduce.invokeStatic(core.clj:6899)
	at clojure.core$mapv.invokeStatic(core.clj:6984)
	at clojure.core$mapv.invoke(core.clj:6984)
	at datomic.common$pooled_mapv.invokeStatic(common.clj:700)
	at datomic.common$pooled_mapv.invoke(common.clj:695)
	at datomic.datalog$qmapv.invokeStatic(datalog.clj:52)
	at datomic.datalog$qmapv.invoke(datalog.clj:47)
	at datomic.datalog.FnRel.join_project_with(datalog.clj:667)
	at datomic.datalog$join_project_coll.invokeStatic(datalog.clj:130)
	at datomic.datalog$join_project_coll.invoke(datalog.clj:128)
	at datomic.datalog$fn__5306.invokeStatic(datalog.clj:233)
	at datomic.datalog$fn__5306.invoke(datalog.clj:229)
	at datomic.datalog$fn __5216$G__ 5209__5231.invoke(datalog.clj:65)
	at datomic.datalog$eval_clause$fn__5843.invoke(datalog.clj:1401)
	... 160 common frames omitted
Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.lang.Thread.start0(Native Method)

```

---

<div class="post-metadata">

**Author:** ![jaret](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/jaret/32/45_2.png) [@jaret](https://forum.datomic.com/u/jaret)\
**Post date:** [June 30, 2023, 11:17am UTC](https://forum.datomic.com/t/outofmemoryerror-when-heavy-data-query-is-executed/2280/2 "2023-06-30T11:17:14Z")

</div>

@nikolayandr can you confirm that the peer application ran out of memory by reviewing metrics of memory on the peer? Can you share the query you believe is causing this issue and the method you are calling along with an estimate for the number of entities this is called for? What are your peer memory settings?

---

<div class="post-metadata">

**Author:** ![nikolayandr](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/nikolayandr/32/375_2.png) [@nikolayandr](https://forum.datomic.com/u/nikolayandr)\
**Post date:** [June 30, 2023, 12:18pm UTC](https://forum.datomic.com/t/outofmemoryerror-when-heavy-data-query-is-executed/2280/3 "2023-06-30T12:18:09Z")

</div>

@jaret  
I can provide only the query now:

```auto
[:find 
?ViaxSubscriptionContract 
:in $ ?ViaxSubscriptionContract_bpStatus_value_4 [?ViaxSubscriptionContract_value_5 ...] 
:where 

;; Produces ~10000 elements
	[?ViaxSubscriptionContract :Model/typeName ?ViaxSubscriptionContract_value_5]

;; io.viax.core.model.query.impl.DynamicFieldValueQueryResolver/getValue executes one http call for each entity
	[(io.viax.core.model.query.impl.DynamicFieldValueQueryResolver/getValue $ ?ViaxSubscriptionContract “bpStatus”) ?ViaxSubscriptionContract_bpStatus_compare_value_3]
	
	[(= ?ViaxSubscriptionContract_bpStatus_compare_value_3 ?ViaxSubscriptionContract_bpStatus_value_4)]
]

```

Unfortunately, I can’t provide the metrics now, but in general, the application becomes unreachable once this error happens so looks like resources are exhausted and the application can’t process the future requests

---

<div class="post-metadata">

**Author:** ![jaret](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/jaret/32/45_2.png) [@jaret](https://forum.datomic.com/u/jaret)\
**Post date:** [June 30, 2023, 1:32pm UTC](https://forum.datomic.com/t/outofmemoryerror-when-heavy-data-query-is-executed/2280/4 "2023-06-30T13:32:40Z")

</div>

> Unfortunately, I can’t provide the metrics now, but in general, the application becomes unreachable once this error happens so looks like resources are exhausted and the application can’t process the future requests

That’s consistent with OOM behavior. Would still be good to capture the metrics so you are sure that is the problem.

@nikolayandr and what is the heap you are giving your application?

Have you tried adding more memory? Alternatively, is there a way to run a smaller query result by splitting the query up?