# Message Listeners in Ions

**URL:** <https://forum.datomic.com/t/message-listeners-in-ions/860>\
**Category:** Datomic Cloud\
**Created:** [February 25, 2019, 7:22pm UTC](https://forum.datomic.com/t/message-listeners-in-ions/860 "2019-02-25T19:22:08Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![okocim](https://avatars.discourse-cdn.com/v4/letter/o/77aa72/32.png) [@okocim](https://forum.datomic.com/u/okocim)\
**Post date:** [February 25, 2019, 7:22pm UTC](https://forum.datomic.com/t/message-listeners-in-ions/860/1 "2019-02-25T19:22:08Z")

</div>

If I have an SQS message listener that I need to run for async processing, is there any way I can start this from within the nodes on a compute group, or do I always have to invoke a lambda from the outside to do such a thing?

I understand that it may not be a good idea to run asynchronous processes inside my compute group, but I’d like to get the benefit of “effective in-memory access to datoms” for the purposes of processing these messages.

---

<div class="post-metadata">

**Author:** ![marshall](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/marshall/32/48_2.png) [@marshall](https://forum.datomic.com/u/marshall)\
**Post date:** [February 26, 2019, 2:12pm UTC](https://forum.datomic.com/t/message-listeners-in-ions/860/2 "2019-02-26T14:12:32Z")

</div>

You can certainly have a background thread that runs on your Datomic node(s) and handles async processing.

See [Kafka Consumer as an Ion?](https://forum.datomic.com/t/kafka-consumer-as-an-ion/823/4?u=marshall)

The main point I should make here is that your process should be coordinated via ion calls, but you should not make individual ion calls that are long-lived (i.e. don’t make `run-my-whole-import-job` a lambda-invoked ion).

---

<div class="post-metadata">

**Author:** ![okocim](https://avatars.discourse-cdn.com/v4/letter/o/77aa72/32.png) [@okocim](https://forum.datomic.com/u/okocim)\
**Post date:** [February 26, 2019, 3:01pm UTC](https://forum.datomic.com/t/message-listeners-in-ions/860/3 "2019-02-26T15:01:28Z")

</div>

Good to know. That still leaves me wondering how to ensure that the message listener is started after every redeploy.

I’m still not completely familiar with the internals of ions, so the crux of my question is really where in the code I should put the bootstrap of such listeners. Normally, I’d do something like this in an application’s “main” entry point and bootstrap the entire system (à la component, mount, integrant, etc.), but I couldn’t figure out how to declare such an entry point with an ion deployment.

Triggering something from the outside for each member of the ASG seems like it would be needlessly complicated.

---

<div class="post-metadata">

**Author:** ![furkan3ayraktar](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/furkan3ayraktar/32/507_2.png) [@furkan3ayraktar](https://forum.datomic.com/u/furkan3ayraktar)\
**Post date:** [March 22, 2021, 4:16pm UTC](https://forum.datomic.com/t/message-listeners-in-ions/860/4 "2021-03-22T16:16:55Z")

</div>

Hi!

I have a relevant question in the topic and would like to hear if you already found a solution.

My understanding is, it is okay to run background threads as long as it is coordinated through Lambda calls. However, this leaves me with another question.

Let’s say I have a Lambda ion that starts the background thread after a deployment. I can trigger that Lambda function via an EventBridge rule that watches the Datomic Ion deployments on CodeDeploy. This way, it will be certain that the Lambda function run after each successful deployment and the background thread will be started. However, if I have one query group with let’s say three nodes where each node should start a background thread. If I’m understanding correctly, the Lambda ion call will only be executed on one of the nodes in the query group rather than all of them. Is there a specific Lambda ion type that executes in all of the nodes or am I missing something else?

Cheers,  
Furkan

---

<div class="post-metadata">

**Author:** ![jmshelby](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/jmshelby/32/437_2.png) [@jmshelby](https://forum.datomic.com/u/jmshelby)\
**Post date:** [May 20, 2021, 4:51pm UTC](https://forum.datomic.com/t/message-listeners-in-ions/860/5 "2021-05-20T16:51:21Z")

</div>

I’m curious why you haven’t just wired up your SQS handler to an ion lambda, and then in AWS just configure the SQS queue to be consumed using that lambda as the handler. This is the main way ions was designed to easily connect with the rest of the AWS ecosystem. The lambda that is created is just a small invoker of your handler inside of the ion, so you’re still getting the in-memory access to datoms. In other words, your handler is still running in the compute group, wether it’s invoked via the lambda, or you run your own async processor.

---

<div class="post-metadata">

**Author:** ![furkan3ayraktar](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/furkan3ayraktar/32/507_2.png) [@furkan3ayraktar](https://forum.datomic.com/u/furkan3ayraktar)\
**Post date:** [June 16, 2021, 7:34am UTC](https://forum.datomic.com/t/message-listeners-in-ions/860/6 "2021-06-16T07:34:51Z")

</div>

Hi @jmshelby! You can check out [this discussion](https://clojurians-log.clojureverse.org/datomic/2021-03-24) at datomic channel in the Clojurians Slack regarding this topic. Also refer to [this blog post](https://zaccharles.medium.com/lambda-concurrency-limits-and-sqs-triggers-dont-mix-well-sometimes-eb23d90122e0) to see the issues using Lambda with SQS polling.

---

<div class="post-metadata">

**Author:** ![stuartrexking](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/stuartrexking/32/496_2.png) [@stuartrexking](https://forum.datomic.com/u/stuartrexking)\
**Post date:** [September 16, 2021, 1:39am UTC](https://forum.datomic.com/t/message-listeners-in-ions/860/7 "2021-09-16T01:39:34Z")

</div>

@marshall This. Is there a solution to this problem for Ion applications?

---

<div class="post-metadata">

**Author:** ![cch1](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/cch1/32/470_2.png) [@cch1](https://forum.datomic.com/u/cch1)\
**Post date:** [February 20, 2024, 10:35pm UTC](https://forum.datomic.com/t/message-listeners-in-ions/860/8 "2024-02-20T22:35:03Z")

</div>

[I know this topic is old, but re-reading it more carefully, I find that @furkan3ayraktar 's (and @stuartrexking and @okocim) question could be expressed as “how can I ensure a startup process runs on each of N autoscaled Ions?”]

Technically, I don’t know of a way to solve the problem. But practically I have been satisfied keeping all my lambda entry points (http-direct and lambdas) in the same namespace and having them close over a single delay that initializes the system (which can include starting a “background worker thread”). The one “startup” delay is de-referenced in each of the entry point handlers. It’s true that some stimulus must arrive on an entry point to start the thread, but if nothing every arrives in the form of “real work” stimulus (e.g. an HTTP request or an SQS message or …) do you even care? I think in most scenarios the answer is no, but I can imagine cases where the goal is to bootstrap many ion instances to chew through a lot of pre-configured work with no “input” -and this approach won’t address that need.

---

<div class="post-metadata">

**Author:** ![onetom](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/onetom/32/430_2.png) [@onetom](https://forum.datomic.com/u/onetom)\
**Post date:** [March 5, 2024, 3:20am UTC](https://forum.datomic.com/t/message-listeners-in-ions/860/9 "2024-03-05T03:20:41Z")

</div>

we are using the same approach too, except, instead of a `clojure.core/delay`, we are using a [`redelay`](https://github.com/aroemers/redelay), which allows us to even re-initialize our service.

since we also start an nREPL server and open up a socket REPL port on our ion instances, it’s possible to modify the function, which constructs our ring router, without redeployment.  
but then , which is baked into the `:http-direct :handler-fn` (specified in the `ion-config.edn`)

```clojure
(ns our.ion
  (:require
    [datomic.ion :as ion]
    ,,,
    [redelay.core :as redelay])
  (:import
    (redelay.core State)))

(def env (-> (ion/get-env) :env (or :local)))
(def replion (replion/start env)) #_(.close replion)

(defonce ^State cfg-ref (redelay/state (api.env/cfg env)))
(defn cfg [] (deref cfg-ref))

(defonce ^State svc-ref (redelay/state (svc/mk (cfg))))
(defn svc [] (deref svc-ref))

(defonce ^State ring-handler-ref
  (redelay/state (ring-api/mk-handler cfg-ref svc-ref)))

(defn ring-handler [req] (@ring-handler-ref req))

```

then, if we want to re-read the config from SSM or want to recompute our ring router, we can eval the following thru our ion REPL (aka [REPLion](https://github.com/markbastian/replion)) connection:

```clojure
  (do (.close cfg-ref)
      (.close svc-ref)
      (.close ring-handler-ref))

```

This approach has the benefit of making this `our.ion` NS always loadable, even if the initialization functions (`api.env/cfg`, `svc/mk`, `ring-api/mk-handler`) would fail.

Since we initialize a REPL server early on, we can debug such failures through a REPL client.

If this NS would fail to load, then it’s retried a few times, then a CodeDeploy rollback gets triggered. Including the deployment time, it could take 5-10minutes to get back to a working state.
