Cannot allocate memory on Ion push

Datomic Cloud
1

We frequently (daily) get an error in our dev env when pushing:

[stderr]OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000fa900000, 41418752, 0) failed; error='Cannot allocate memory' (errno=12)
[stdout]# There is insufficient memory for the Java Runtime Environment to continue.
[stdout]# Native memory allocation (mmap) failed to map 41418752 bytes for committing reserved memory.
[stdout]# An error report file with more information is saved as:
[stdout]# /tmp/hs_err_pid14747.log

Looks like the java process is using 1639896 bytes but our heap usage is

Heap Configuration:
Heap Usage:
G1 Heap:
    used     = 314203648 (299.64794921875MB)
G1 Young Generation:
Eden Space:
   used     = 126877696 (121.0MB)
Survivor Space:
   used     = 15728640 (15.0MB)
G1 Old Generation:
   used     = 171597312 (163.64794921875MB)

Is there a way of altering the default settings of -Xmx1220m -Xms1220m? (we’d probs be better at 800/900Mb)

2

Forgot to say: solo deployment.

3

Hi Dean,

Solo runs on a very small instance with limited memory (by design, to keep the cost low). The error you’re seeing indicates that the deployment you’re running exceeds the memory available for a solo system. When you need more capability, you can upgrade to Production.

Thanks,
Jaret

4

Sure, we already use production for our production site - this is our test system. We’d like to use more test systems, and obviously using prod deployments will increase the cost dramatically. The application isn’t under memory pressure but it seems the jvm is taking up more native memory than is possible for the deployment to run.

1 Like
5

I believe the official recommendation (as per https://docs.datomic.com/cloud/operation/planning.html#stages) is to use separate query groups for testing rather than setting up entirely new systems.

6

@Dean I discussed your case with the Product team this morning and we’re interested in learning more as we havent seen this profile of mem usage in our testing. I am going to put my request here in the forum post, but it might be prudent to move this discussion to a support case where we can more easily share information and credentials. If that’s suitable to you, shoot me an e-mail at support@cognitect.com to create a case.

Questions we have:

  1. What jdk invocation is failing? (The deploy or the actual process)
  2. When does this happen more specifically? I know you mentioned on-push, but is it every push?
  3. What generally does your program do? Is there anything unique? Are you setting memory in your program?
  4. Finally, if you’re fine to open a case we would like to get log access to look at this system. The access would be Read-only and I can share details on how to create the account in the case.

Thanks,
Jaret

7

Hi Jaret, I sent a mail (from personal account linked here and business account) but didn’t get a response, has it been lost in spam?

Cheers,
Dean

8

Hi @Dean it appears it did get caught in spam. I am recovering and will update the case shortly. Apologies for the delay!

1 Like
9

For other people this affects, it was fixed by an upgrade to ion-dev (0.9.265) - thanks to the datomic team.

1 Like
10

I ran into this issue today running ion-dev (0.9.282). The solution from this thread, while not good did allow me to move forward. The solution is to reboot/terminate the solo EC2 instance.