# Is it possible to have a 'worker application' as an Ion? **URL:** \ **Category:** Datomic Cloud\ **Created:** [July 1, 2021, 9:35am UTC](https://forum.datomic.com/t/is-it-possible-to-have-a-worker-application-as-an-ion/1887 "2021-07-01T09:35:03Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![pedrorgirardi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/pedrorgirardi/32/498_2.png) [@pedrorgirardi](https://forum.datomic.com/u/pedrorgirardi)\ **Post date:** [July 1, 2021, 9:35am UTC](https://forum.datomic.com/t/is-it-possible-to-have-a-worker-application-as-an-ion/1887/1 "2021-07-01T09:35:03Z")
I have a Clojure application that consumes messages from an MQTT broker and transacts them into Datomic. Is it possible to implement this use case with Ions? We’re migrating an application to Datomic, and we would like to use Ions as much as possible. ---
**Author:** ![eraad](https://avatars.discourse-cdn.com/v4/letter/e/b77776/32.png) [@eraad](https://forum.datomic.com/u/eraad)\ **Post date:** [July 1, 2021, 10:12pm UTC](https://forum.datomic.com/t/is-it-possible-to-have-a-worker-application-as-an-ion/1887/2 "2021-07-01T22:12:07Z")
Hi Pedro, we implemented this pattern using Ions subscribed directly to an AWS SQS queue. Each time there is a new message, the Ion gets executed and the message is processed. Ions are basically Lambdas so you can run them either periodically or triggered by AWS supported events (SQS, S3, etc.) The only issue with this pattern is that the SQS trigger provided by AWS is not configurable in terms of concurrency. Depending on your requirements it may or may not be adequate. Check this article for more information about it: > **[Lambda Concurrency Limits and SQS Triggers Don’t Mix Well (Sometimes)](https://zaccharles.medium.com/lambda-concurrency-limits-and-sqs-triggers-dont-mix-well-sometimes-eb23d90122e0)** > > SQS can be used to trigger Lambda functions. Awesome! But it’s not all sunshine and roses… We have had trouble with it, so we are probably moving this to a Clojure process running on a machine that listens to SQS and processes the messages directly, without dealing with Lambda. ---
**Author:** ![pedrorgirardi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/pedrorgirardi/32/498_2.png) [@pedrorgirardi](https://forum.datomic.com/u/pedrorgirardi)\ **Post date:** [July 2, 2021, 4:48am UTC](https://forum.datomic.com/t/is-it-possible-to-have-a-worker-application-as-an-ion/1887/3 "2021-07-02T04:48:35Z")
Hi Eduardo, it’s an MQTT broker and not managed by AWS so I can’t connect it directly to an Ion - I think. But it’s cool to know about SQS, so thanks for the link! I was thinking about deploying this program in an EC2 instance or maybe Beanstalk, subscribe to the topic and publish to an SQS queue, or something else, which my Ion would process. What issues did you have with this setup? Maybe it’s simpler without Ion in this case? I might be being silly forcing myself to use Ion for this case. ---
**Author:** ![furkan3ayraktar](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/furkan3ayraktar/32/507_2.png) [@furkan3ayraktar](https://forum.datomic.com/u/furkan3ayraktar)\ **Post date:** [July 2, 2021, 9:55am UTC](https://forum.datomic.com/t/is-it-possible-to-have-a-worker-application-as-an-ion/1887/4 "2021-07-02T09:55:46Z")
We use SQS but not through Lambda’s. We start a background thread when the Ion is deployed which uses long polling for messages. There was a discussion [here on this thread](https://forum.datomic.com/t/message-listeners-in-ions/860/2) about message listeners on Ions. ---
**Author:** ![pedrorgirardi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/pedrorgirardi/32/498_2.png) [@pedrorgirardi](https://forum.datomic.com/u/pedrorgirardi)\ **Post date:** [July 2, 2021, 10:44am UTC](https://forum.datomic.com/t/is-it-possible-to-have-a-worker-application-as-an-ion/1887/5 "2021-07-02T10:44:59Z")
The other discussions were helpful, thanks @furkan3ayraktar. I’m getting started with Datomic Ions, and I’m trying to understand Marshall’s response: > You can certainly have a background thread that runs on your Datomic node(s) and handles async processing. This ‘background thread that runs on your Datomic’ is a background thread that was started inside an Ion? And your approach to starting the background thread: > Let’s say I have a Lambda ion that starts the background thread after a deployment. I can trigger that Lambda function via an EventBridge rule that watches the Datomic Ion deployments on CodeDeploy. Is this how you’re doing it? Would you mind giving more details about how you did it? ---
**Author:** ![furkan3ayraktar](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/furkan3ayraktar/32/507_2.png) [@furkan3ayraktar](https://forum.datomic.com/u/furkan3ayraktar)\ **Post date:** [July 4, 2021, 11:44am UTC](https://forum.datomic.com/t/is-it-possible-to-have-a-worker-application-as-an-ion/1887/6 "2021-07-04T11:44:14Z")
Hi @pedrorgirardi, I can try to explain more in detail. You can start a background thread with using [future](https://clojuredocs.org/clojure.core/future) and poll your queue within that thread. See below for a simplified version of the function we use to start the background thread and poll for messages from SQS: ```auto (defn listen-messages! [status queue-url] (future (try (loop [] (reset! status :listening) (let [messages (receive-messages queue-url)] ; Long polling SQS for new messages (doseq [message messages] (handle-message! message))) (if (not= :listening @status) nil (recur))) ;; InterruptedException happens if the future is cancelled with future-cancel. (catch InterruptedException _ (reset! status :stopped) nil) ;; Catch and handle any other unknown exceptions. (catch Exception e (handle-consumer-exception! "Unknown exception in consumer listener." e))))) (def queue-status (atom :initial)) (listen-messages! queue-status "https://sqs.eu-west-1.amazonaws.com/XXXXXXXXXXXX/your-queue-name") ``` The function above is called right away so that it will be executed as soon as Datomic Ion deployment loads that namespace. At the beginning, we wanted to control the start/stop logic of this background thread, to have more control over it. That’s why I had a EventBridge rule that was watching the successful Datomic Ion deployments on CloudDeploy. However, I noticed that, triggering that Lambda will only be executed in one of the many auto-scaled instances and as a result will only start the background thread on one of them, instead of all. In order to achieve what we wanted, we needed to write more complex logic so we skipped it for now and decided to run the threads right away. We added some monitoring logic which notifies us in case the background thread dies for a reason, which did not happen yet. ---
**Author:** ![pedrorgirardi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/pedrorgirardi/32/498_2.png) [@pedrorgirardi](https://forum.datomic.com/u/pedrorgirardi)\ **Post date:** [July 5, 2021, 2:34am UTC](https://forum.datomic.com/t/is-it-possible-to-have-a-worker-application-as-an-ion/1887/7 "2021-07-05T02:34:01Z")
Thank you very much @furkan3ayraktar ! It’s super clear what you mean now and how I could do it in my project.