While running high-throughput write/load tests against the latest version of Datomic 1.0.7364 (on-prem), we’re consistently seeing this warning in the logs:
WARN | ... | AMQ212054: Destination address=viax-viax-bcore-...tx-submit is blocked. If the system is configured to block make sure you consume messages on this configuration.
It appears to originate from the messaging layer (ActiveMQ Artemis), possibly related to internal Datomic tx submission queues.
One possible cause I found online is low disk space on the broker or system, but in our case, disk space is not an issue — there’s plenty available.
Questions:
Is this warning expected under load?
Does it indicate a real problem (e.g., slow consumers or backpressure)?
Should we take any specific action to avoid it or mitigate potential issues?
That message indicates that the Artemis tx-submit queue is full. It means that you are submitting faster than you are transacting, but it can be safely ignored.
For monitoring load tests we recommend relying on Alerts, exceptions, and the process monitor metrics in the logs. The Artemis warnings can be ignored.
Do you observe any issues in load testing which you would like to discuss? We can help you transact faster, but suspect that would require a support ticket to share details about your system and your test. If thats something you are interested in please e-mail support@datomic.com.
In our case, we’re simulating 100 concurrent users, each executing synchronous transactions using datomic.api/transact. That means up to 100 transactions can be in-flight at the same time.
Could this level of concurrency be enough to trigger the Artemis “out of credit” warning? Or would you consider this a normal load that shouldn’t typically result in such a message?
This warning is expected under load. It does not represent a real problem and specifically you do not need to take any actions based on this warning.
Could this level of concurrency be enough to trigger the Artemis “out of credit” warning?
You are submitting transactions to a “queue” faster than the transactor can dequeue them. Artemis doesn’t touch disk, its an in-memory pipe. You should focus your attention on reducing work per transaction by using Hints, tx-stats, io-stats, or looking for AlarmBackpressure where you are bound by indexing.
Also happy to look at the results of your simulation and Datomic level problems you might be encountering? Is there something after this warning that you are seeing or not seeing that you would expect? Or put another way, ignoring the warning how does your simulation behave?