Performance degradation and heap overflow with large collection binding input

Datomic Pro Troubleshooting
1

Hi everyone,

I’m working on a query pattern where I pass a large number of entity IDs into the query as a collection binding. Here’s a simplified version of the query:

[:find ?MaterialImpl1
 :in $ [?MaterialImpl1 ...] ?MaterialImpl1_value_0_VR
 :where
 [?MaterialImpl1 :Model/typeName ?MaterialImpl1_value_0_VR]]

When the number of IDs is relatively small (e.g., up to 100,000), the query executes within a reasonable time (a few seconds). However, once the input size increases (e.g., around 1,000,000 IDs), I observe a drastic performance slowdown. Eventually, the query fails with a Java heap space overflow, which I assume is due to memory pressure during query execution.

I’m trying to understand: Is there a known upper limit or best practice for using large collection bindings like [?e ...]?

Any advice or experiences would be appreciated — thank you!

Query performance with large(r) inputs
2

Additionally, I’ve noticed that if we run a query like this:

[:find ?MaterialImpl1
 :in $ ?MaterialImpl1_value_0_VR
 :where
  [?_ :someAttribute ?MaterialImpl1]
  [?MaterialImpl1 :Model/typeName ?MaterialImpl1_value_0_VR]]

where ?MaterialImpl1 ends up containing exactly the same set of entity IDs as in the problematic case, the query executes much faster and is more memory-efficient.

The issue is that it’s not always possible to construct the query this way, since the set of entity IDs bound to ?MaterialImpl1 may come from multiple places in the application logic, not just from a single [?_ :someAttribute ?MaterialImpl1] clause. In those cases, we are forced to pass a large collection as an input binding, which causes the performance degradation and memory pressure described in this thread.

@jaret I’d really appreciate it if you could confirm whether an efficient solution for this problem exists.

P.S. It’s quite surprising that a query operating over the exact same data set can have such a significant difference in performance.

3

Can you run the query with :query-stats and :io-stats and share the result?

(-> (d/query {:query ... :args ... :query-stats true :io-context :user/ctx})
      (dissoc :ret))
4

I’m sorry, my mistake. I tested in debug mode, and apparently, when I executed a query with about one million elements in debug mode, it took much longer for some reason than when I performed the same action in normal mode without debugging. Thank you for your attention.

5

@nikolayandr do you mean your IDE’s debug mode? I am just curious about the specifics so I am sure I understand.

6

Yes, that’s correct — I was testing in debug mode using IntelliJ and checked both scenarios mentioned in my post. The one where the query accepted around a million records was indeed significantly slower. However, when I prepared a clean and minimal example in the REPL to share with you, everything turned out to work fast.