Kafka Consumer as an Ion?

Datomic Cloud
1

I would like to consume Kafka topics in the Datomic Cloud cluster.

Is that advisable/possible? Naively I’ll just do this in an Ion:

(async/thread-call consume-from-topic-forever! kafka-consumer kafka-topic)

1 Like
2

Nothing stopping you. :wink:

3

Generally I would recommend against using an Ion for a long-running process. Ions are designed to execute and return within the timeout of an API Gateway or AWS Lambda call.
As solussd mentions, there is nothing in Datomic that will prevent you from doing this, but your thread will be out there running without any way for you to access or control it and potentially competing with the Datomic system for resources/etc.

4

Danie,

I’ve discussed this approach further with the team, and want to amend my prior statement.

You certainly can have a long-lived background process that does a job (i.e. kafka consumer) via a thread in Datomic. The specific pattern to avoid would be a long running process that lives inside an ion function call (i.e. an ion function that is my-full-import).

As with any long-running process you do need to be aware of and design around how you’ll handle communication with the background job, how you’ll manage any required state, etc.

5

Any suggestions/thoughts on the best way to initialize/manage these kinds of processes? Say at read-time vs I dunno, an ion/lambda that can start stop something that lives on another thread?

6

Probably a related question: Can we have a Quarzite or a scheduler long-running process as an ion? planning to design the start/stop/re-start from a web page.

7

You’d definitely want to use an Ion to trigger start/stop/etc.

8

I’m not specifically familiar with Quartzite, but for scheduling things I would generally use AWS’ lambda triggers (i.e. https://docs.aws.amazon.com/lambda/latest/dg/tutorial-scheduled-events-schedule-expressions.html)

9

Thank you for the response. I will look into it. Haven’t thought about it from this angle!

10

@marshall How is this supposed to work in a compute group with multiple nodes? If you use an Ion to start a process (like a Kafka consumer), it only starts on one node, not all of them.

We’ve just encountered this exact problem. There isn’t a way to reliably start and stop (manage) processes with Ions. We are using Kafka and want to manage the consumers, but it doesn’t look like it’s possible from an Ions node, so we are going to move away from Ions.

Do you know if there is a plan for lifecycle management for Ion applications? There should be some definition in the ion-config.edn for starting and stopping things.

1 Like
11

Hi @stuartrexking assuming the lambda trigger suggestion wasn’t the solution you were looking for perhaps simply leveraging namespace loading could be a path forward. Maybe invoking some type of startup-hook function in one of your core namespaces could be used to kick off needed processes.