# Jetty max threads error when enabling ping /health

**URL:** <https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603>\
**Category:** Troubleshooting\
**Created:** [September 5, 2018, 10:04am UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603 "2018-09-05T10:04:34Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![nestrada](https://avatars.discourse-cdn.com/v4/letter/n/3bc359/32.png) [@nestrada](https://forum.datomic.com/u/nestrada)\
**Post date:** [September 5, 2018, 10:04am UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/1 "2018-09-05T10:04:35Z")

</div>

Hello, I’m running datomic pro within a Kubernetes cluster and I wanted to set up a liveness probe using the /health HTTP endpoint described here :

> java.lang.RuntimeException: Unable to start ping endpoint localhost:9999  
> at datomic.transactor\_ext$start\_ping\_endpoint.invokeStatic(transactor\_ext.clj:56)  
> at datomic.transactor\_ext$start\_ping\_endpoint.invoke(transactor\_ext.clj:39)  
> at datomic.transactor\_ext$start\_pro.invokeStatic(transactor\_ext.clj:63)  
> at datomic.transactor\_ext$start\_pro.invoke(transactor\_ext.clj:59)  
> at clojure.lang.Var.invoke(Var.java:381)  
> at datomic.transactor$run\_STAR\_.invokeStatic(transactor.clj:303)  
> at datomic.transactor$run\_STAR\_.invoke(transactor.clj:234)  
> at datomic.transactor$run$fn\_\_26995.invoke(transactor.clj:358)  
> at clojure.core$binding\_conveyor\_fn$fn\_\_5476.invoke(core.clj:2022)  
> at clojure.lang.AFn.call(AFn.java:18)  
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
> at java.lang.Thread.run(Thread.java:748)  
> Caused by: java.lang.IllegalStateException: Insufficient threads: max=3 \< needed(acceptors=1 + selectors=2 + request=1)  
> at org.eclipse.jetty.server.Server.doStart(Server.java:368)  
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)  
> at cognitect.http\_endpoint.jetty$start.invokeStatic(jetty.clj:123)  
> at cognitect.http\_endpoint.jetty$start.invoke(jetty.clj:120)  
> at cognitect.http\_endpoint.jetty$server$fn\_\_9606.invoke(jetty.clj:136)  
> at cognitect.http\_endpoint$create\_endpoint.invokeStatic(http\_endpoint.clj:306)  
> at cognitect.http\_endpoint$create\_endpoint.invoke(http\_endpoint.clj:231)  
> at cognitect.nano\_impl.server$create.invokeStatic(server.clj:221)  
> at cognitect.nano\_impl.server$create.invoke(server.clj:187)  
> at cognitect.nano\_impl$create.invokeStatic(nano\_impl.clj:228)  
> at cognitect.nano\_impl$create.invoke(nano\_impl.clj:152)  
> at datomic.transactor\_ext$start\_ping\_endpoint.invokeStatic(transactor\_ext.clj:47)

But I’m getting this exception and I can’t seem to find a workaround (system property, environment variable etc. in order to tune the Jetty thread pool)

I’m running the latest version of datomic-pro 0.9.5703. Any suggestions on how I should proceed? Thanks in advance

---

<div class="post-metadata">

**Author:** ![nestrada](https://avatars.discourse-cdn.com/v4/letter/n/3bc359/32.png) [@nestrada](https://forum.datomic.com/u/nestrada)\
**Post date:** [September 26, 2018, 4:31pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/2 "2018-09-26T16:31:58Z")

</div>

Anybody have any insight on this? Is on-prem just not used anymore or is no longer supported?

---

<div class="post-metadata">

**Author:** ![Folcon](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/folcon/32/175_2.png) [@Folcon](https://forum.datomic.com/u/Folcon)\
**Post date:** [October 29, 2018, 10:51am UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/3 "2018-10-29T10:51:30Z")

</div>

Hey @nestrada, I’m using datomic-pro on-prem, I’ve actually not had to setup the heartbeat step you’re running.  
Any way I could give you a hand?

---

<div class="post-metadata">

**Author:** ![nestrada](https://avatars.discourse-cdn.com/v4/letter/n/3bc359/32.png) [@nestrada](https://forum.datomic.com/u/nestrada)\
**Post date:** [October 30, 2018, 2:03pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/4 "2018-10-30T14:03:41Z")

</div>

Hey @Folcon, I also able to run the transactor on-prem without k8s heartbeat configured, but our notification system likes to be able to detect failures so that we can investigate later. I’m simply trying to enable the ping endpoint described here [Health Check Endpoint](https://docs.datomic.com/on-prem/transactor.html#health-check-endpoint).

Is anyone able to get this working? Thanks for any help

---

<div class="post-metadata">

**Author:** ![Folcon](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/folcon/32/175_2.png) [@Folcon](https://forum.datomic.com/u/Folcon)\
**Post date:** [October 30, 2018, 5:53pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/5 "2018-10-30T17:53:24Z")

</div>

Reading that stacktrace, any way you can increase the threadpool size in jetty?

```auto
Caused by: java.lang.IllegalStateException: Insufficient threads: max=3 < needed(acceptors=1 + selectors=2 + request=1)
at org.eclipse.jetty.server.Server.doStart(Server.java:368)

```

That seems to be the main problem.

I’ve never run into this, are you limiting the pool size in some way?

---

<div class="post-metadata">

**Author:** ![nestrada](https://avatars.discourse-cdn.com/v4/letter/n/3bc359/32.png) [@nestrada](https://forum.datomic.com/u/nestrada)\
**Post date:** [November 7, 2018, 11:32am UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/6 "2018-11-07T11:32:50Z")

</div>

I’m not doing anything besides running the transactor with the provided scripts and these lines added in my transactor.properties file.

```
ping-host=localhost
ping-port=9999

```

Try it out and you’ll see (using latest datomic-pro btw)

---

<div class="post-metadata">

**Author:** ![marshall](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/marshall/32/48_2.png) [@marshall](https://forum.datomic.com/u/marshall)\
**Post date:** [November 8, 2018, 3:17pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/7 "2018-11-08T15:17:19Z")

</div>

Would it be possible for you to provide the output of running:

`(.availableProcessors (java.lang.Runtime/getRuntime))`

from a REPL in the environment where you’re hitting this issue?

---

<div class="post-metadata">

**Author:** ![nestrada](https://avatars.discourse-cdn.com/v4/letter/n/3bc359/32.png) [@nestrada](https://forum.datomic.com/u/nestrada)\
**Post date:** [November 12, 2018, 6:00pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/8 "2018-11-12T18:00:45Z")

</div>

I don’t know of any way to run the transactor from a REPL. However you may be onto something as I’m running the transactor within a docker container running on jdk8.

---

<div class="post-metadata">

**Author:** ![marshall](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/marshall/32/48_2.png) [@marshall](https://forum.datomic.com/u/marshall)\
**Post date:** [November 12, 2018, 6:27pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/9 "2018-11-12T18:27:05Z")

</div>

I’ve been unable to reproduce the issue. I’ve tried on instances with 1 core, up to \> 12 cores and in all cases I’m able to start the transactor with the `ping-host` and `ping-port` specified in the properties file and then I’m able to curl that endpoint successfully.

---

<div class="post-metadata">

**Author:** ![nestrada](https://avatars.discourse-cdn.com/v4/letter/n/3bc359/32.png) [@nestrada](https://forum.datomic.com/u/nestrada)\
**Post date:** [November 14, 2018, 5:55pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/10 "2018-11-14T17:55:04Z")

</div>

```auto
    
    Executing : /docker-java-home/bin/java -server -showversion -XshowSettings:vm -XX:+PrintCommandLineFlags -XX:+ExitOnOutOfMemoryError -XX:+UseStringDeduplication -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XX:+UseG1GC -XX:MaxGCPauseMillis=50 -Xms4g -Xmx4g -Ddatomic.pingHost=localhost -Ddatomic.pingPort=9999 -classpath resources:lib/*:datomic-transactor-pro-0.9.5783.jar:samples/clj:bin: clojure.main --main datomic.launcher /oscaro/etc/transactor.properties
    -XX:+ExitOnOutOfMemoryError -XX:InitialHeapSize=4294967296 -XX:MaxGCPauseMillis=50 -XX:MaxHeapSize=4294967296 -XX:MaxRAMFraction=1 -XX:+PrintCommandLineFlags -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseStringDeduplication 
    VM settings:
        Min. Heap Size: 4.00G
        Max. Heap Size: 4.00G
        Ergonomics Machine Class: server
        Using VM: OpenJDK 64-Bit Server VM
    
    openjdk version "1.8.0_171"
    OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~deb9u1-b11)
    OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)
    
    2018-11-14 17:51:33.166 INFO default org.eclipse.jetty.util.log - Logging initialized @4709ms
    2018-11-14 17:51:35.147 INFO default org.eclipse.jetty.server.Server - jetty-9.3.7.v20160115
    2018-11-14 17:51:35.181 ERROR default datomic.process - {:message "Critical failure, cannot continue: Error starting transactor", :pid 1, :tid 27}
    java.lang.RuntimeException: Unable to start ping endpoint localhost:9999
      at datomic.transactor_ext$start_ping_endpoint.invokeStatic(transactor_ext.clj:56)
      at datomic.transactor_ext$start_ping_endpoint.invoke(transactor_ext.clj:39)
      at datomic.transactor_ext$start_pro.invokeStatic(transactor_ext.clj:63)
      at datomic.transactor_ext$start_pro.invoke(transactor_ext.clj:59)
      at clojure.lang.Var.invoke(Var.java:381)
      at datomic.transactor$run_STAR_.invokeStatic(transactor.clj:305)
      at datomic.transactor$run_STAR_.invoke(transactor.clj:236)
      at datomic.transactor$run$fn__28715.invoke(transactor.clj:362)
      at clojure.core$binding_conveyor_fn$fn__5476.invoke(core.clj:2022)
      at clojure.lang.AFn.call(AFn.java:18)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.IllegalStateException: Insufficient threads: max=3 < needed(acceptors=1 + selectors=2 + request=1)
      at org.eclipse.jetty.server.Server.doStart(Server.java:368)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
      at cognitect.http_endpoint.jetty$start.invokeStatic(jetty.clj:123)
      at cognitect.http_endpoint.jetty$start.invoke(jetty.clj:120)
      at cognitect.http_endpoint.jetty$server$fn__9606.invoke(jetty.clj:136)
      at cognitect.http_endpoint$create_endpoint.invokeStatic(http_endpoint.clj:306)
      at cognitect.http_endpoint$create_endpoint.invoke(http_endpoint.clj:231)
      at cognitect.nano_impl.server$create.invokeStatic(server.clj:221)
      at cognitect.nano_impl.server$create.invoke(server.clj:187)
      at cognitect.nano_impl$create.invokeStatic(nano_impl.clj:228)
      at cognitect.nano_impl$create.invoke(nano_impl.clj:152)
      at datomic.transactor_ext$start_ping_endpoint.invokeStatic(transactor_ext.clj:47)
      ... 13 common frames omitted

```

Here are my command line arguments for running within docker (and kubernetes)

---

<div class="post-metadata">

**Author:** ![marshall](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/marshall/32/48_2.png) [@marshall](https://forum.datomic.com/u/marshall)\
**Post date:** [November 14, 2018, 6:26pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/11 "2018-11-14T18:26:00Z")

</div>

I was able to run the transactor with your same JVM args and use the ping endpoint successfully:

```auto
$ java -server -showversion -XshowSettings:vm -XX:+PrintCommandLineFlags -XX:+ExitOnOutOfMemoryError -XX:+UseStringDeduplication -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XX:+UseG1GC -XX:MaxGCPauseMillis=50 -Xms4g -Xmx4g -Ddatomic.pingHost=localhost -Ddatomic.pingPort=9999 -classpath resources:lib/*:datomic-transactor-pro-0.9.5773.jar:samples/clj:bin: clojure.main --main datomic.launcher ../config/dev.properties
Java HotSpot(TM) 64-Bit Server VM warning: Unable to open cgroup memory limit file /sys/fs/cgroup/memory/memory.limit_in_bytes (No such file or directory)
-XX:+ExitOnOutOfMemoryError -XX:InitialHeapSize=4294967296 -XX:MaxGCPauseMillis=50 -XX:MaxHeapSize=4294967296 -XX:MaxRAMFraction=1 -XX:+PrintCommandLineFlags -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseStringDeduplication
VM settings:
    Min. Heap Size: 4.00G
    Max. Heap Size: 4.00G
    Ergonomics Machine Class: server
    Using VM: Java HotSpot(TM) 64-Bit Server VM

java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)

Starting datomic:dev://localhost:4334/<DB-NAME>, storing data in: data ...
System started datomic:dev://localhost:4334/<DB-NAME>, storing data in: data

```

Ping Endpoint:

```auto
$ curl localhost:9999/health
ok

```

I will continue to experiment on additional VMs, but I would suspect your docker environment.

---

<div class="post-metadata">

**Author:** ![avdelst](https://avatars.discourse-cdn.com/v4/letter/a/f9ae1b/32.png) [@avdelst](https://forum.datomic.com/u/avdelst)\
**Post date:** [November 15, 2018, 2:07pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/12 "2018-11-15T14:07:17Z")

</div>

fwiw, I’m having the same issue when trying to enable the ping healthcheck when running in an openshift pod.

---

<div class="post-metadata">

**Author:** ![milt](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/milt/32/238_2.png) [@milt](https://forum.datomic.com/u/milt)\
**Post date:** [December 7, 2018, 2:33pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/13 "2018-12-07T14:33:52Z")

</div>

@marshall It definitely seems to be container-related. We just ran into this after attempting an upgrade to 0.9.5786. Some more info, collected on a 6-core (12 virtual) 2018 macbook pro:

In the datomic REPL running on the machine (java 8u181):

```
(.availableProcessors (java.lang.Runtime/getRuntime)) ;; => 12

```

In the datomic REPL in the container (java 8u181, Docker Desktop 2.0.0.0-mac81 29211):

```
(.availableProcessors (java.lang.Runtime/getRuntime)) ;; => 6

```

In that same docker environment, only when run with ping host/port set:

```
java.lang.IllegalStateException: Insufficient threads: max=4 < needed(acceptors=1 + selectors=3 + request=1)

```

Without the ping stuff set, works as expected. Also, if I raise docker’s CPU resource limit (in the UI from 6 (the default) to 12, it works with the ping setup. I’ll play with it some more and post back any useful info I find.

---

<div class="post-metadata">

**Author:** ![marshall](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/marshall/32/48_2.png) [@marshall](https://forum.datomic.com/u/marshall)\
**Post date:** [December 7, 2018, 2:48pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/14 "2018-12-07T14:48:27Z")

</div>

Hi Milt,

Thanks for the report.  
We’re continuing to investigate this issue but haven’t yet found a smoking gun.

---

<div class="post-metadata">

**Author:** ![milt](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/milt/32/238_2.png) [@milt](https://forum.datomic.com/u/milt)\
**Post date:** [December 7, 2018, 2:49pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/15 "2018-12-07T14:49:31Z")

</div>

Lol, restricting the CPUs (using docker’s --cpus arg) to 3 or less also seems to resolve the problem (when the global docker setting is at 6, the default)

---

<div class="post-metadata">

**Author:** ![jaret](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/jaret/32/45_2.png) [@jaret](https://forum.datomic.com/u/jaret)\
**Post date:** [December 7, 2018, 2:50pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/16 "2018-12-07T14:50:40Z")

</div>

Hi Milt,

Another user found that having CPU set to 1, 3, and 8 worked. I’d be curious if 8 works in your case.

Thanks,  
Jaret

---

<div class="post-metadata">

**Author:** ![milt](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/milt/32/238_2.png) [@milt](https://forum.datomic.com/u/milt)\
**Post date:** [December 7, 2018, 3:01pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/17 "2018-12-07T15:01:16Z")

</div>

I can confirm, 8 works. So far for me, that’s 1,2,3 and 8+ that work.

---

<div class="post-metadata">

**Author:** ![nestrada](https://avatars.discourse-cdn.com/v4/letter/n/3bc359/32.png) [@nestrada](https://forum.datomic.com/u/nestrada)\
**Post date:** [December 7, 2018, 3:11pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/18 "2018-12-07T15:11:22Z")

</div>

I’m hoping that a solution can be found without resorting to passing `docker run` parameters as I’m running a transactor from Kubernetes and can specify cpu limits but not using docker. Perhaps a system property or environment variable of some sort?

---

<div class="post-metadata">

**Author:** ![cqowsy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/cqowsy/32/110_2.png) [@cqowsy](https://forum.datomic.com/u/cqowsy)\
**Post date:** [January 16, 2019, 11:53pm UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/19 "2019-01-16T23:53:31Z")

</div>

So I just hit this same issue locally. Has anyone had any luck with deployments out to ECS, etc? I changed my allocated cpu to docker desktop to 3 and presto datomic launched with a health endpoint.

---

<div class="post-metadata">

**Author:** ![jaret](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/jaret/32/45_2.png) [@jaret](https://forum.datomic.com/u/jaret)\
**Post date:** [January 22, 2019, 1:25am UTC](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603/20 "2019-01-22T01:25:39Z")

</div>

@cqowsy We are releasing the ability to make the thread calculation configurable along with documentation on how to do so in the next release to resolve this issue. We are working on some other features for that release and so I do not have a time table for when you’ll see it. In the interim, I can share with you configurations (CPU set to 1,3,8) that worked when you have control over the number of CPUS. I know with Docker or Openshift etc you may not have as much control over the max number of threads.

The following configs worked with health endpoint worked:  
openjdk 8, 1 cpu  
openjdk 8, 3 cpu  
oracle 8, 3 cpu  
openjdk 8, 8 cpu  
oracle 8, 8 cpu

The following configs failed:

openjdk 8, 2 cpu  
oracle 8, 2 cpu  
openjdk 8, 4 cpu  
oracle 8, 4 cpu  
openjdk 8, 5 cpu  
oracle 8, 5 cpu  
openjdk 8, 6 cpu  
oracle 8, 6 cpu  
openjdk 8, 7 cpu  
oracle 8, 7 cpu

I’ll be sure to update this thread with a link to the documentation and the new feature once released.

[Next page](https://forum.datomic.com/t/jetty-max-threads-error-when-enabling-ping-health/603.md?page=2)