Kafka Consumer as an Ion?


#1

I would like to consume Kafka topics in the Datomic Cloud cluster.

Is that advisable/possible? Naively I’ll just do this in an Ion:

(async/thread-call consume-from-topic-forever! kafka-consumer kafka-topic)


#2

Nothing stopping you. :wink:


#3

Generally I would recommend against using an Ion for a long-running process. Ions are designed to execute and return within the timeout of an API Gateway or AWS Lambda call.
As solussd mentions, there is nothing in Datomic that will prevent you from doing this, but your thread will be out there running without any way for you to access or control it and potentially competing with the Datomic system for resources/etc.


#4

Danie,

I’ve discussed this approach further with the team, and want to amend my prior statement.

You certainly can have a long-lived background process that does a job (i.e. kafka consumer) via a thread in Datomic. The specific pattern to avoid would be a long running process that lives inside an ion function call (i.e. an ion function that is my-full-import).

As with any long-running process you do need to be aware of and design around how you’ll handle communication with the background job, how you’ll manage any required state, etc.


#5

Any suggestions/thoughts on the best way to initialize/manage these kinds of processes? Say at read-time vs I dunno, an ion/lambda that can start stop something that lives on another thread?