Is it possible to have a 'worker application' as an Ion?

Datomic Cloud
6

Hi @pedrorgirardi, I can try to explain more in detail. You can start a background thread with using future and poll your queue within that thread. See below for a simplified version of the function we use to start the background thread and poll for messages from SQS:

(defn listen-messages! [status queue-url]
  (future
    (try
      (loop []
        (reset! status :listening)
        (let [messages (receive-messages queue-url)] ; Long polling SQS for new messages
          (doseq [message messages]
            (handle-message! message)))
        (if (not= :listening @status)
          nil
          (recur)))

      ;; InterruptedException happens if the future is cancelled with future-cancel.
      (catch InterruptedException _
        (reset! status :stopped)
        nil)

      ;; Catch and handle any other unknown exceptions.
      (catch Exception e
        (handle-consumer-exception! "Unknown exception in consumer listener." e)))))

(def queue-status (atom :initial))

(listen-messages! queue-status "https://sqs.eu-west-1.amazonaws.com/XXXXXXXXXXXX/your-queue-name")

The function above is called right away so that it will be executed as soon as Datomic Ion deployment loads that namespace.

At the beginning, we wanted to control the start/stop logic of this background thread, to have more control over it. That’s why I had a EventBridge rule that was watching the successful Datomic Ion deployments on CloudDeploy. However, I noticed that, triggering that Lambda will only be executed in one of the many auto-scaled instances and as a result will only start the background thread on one of them, instead of all. In order to achieve what we wanted, we needed to write more complex logic so we skipped it for now and decided to run the threads right away. We added some monitoring logic which notifies us in case the background thread dies for a reason, which did not happen yet.

2 Likes