I’m providing a global service and I want to reduce latency.
One way to do this might be to create N cloud dbs in different regions and replicate a users data from the original db to the others. This could be for all data or just on-demand when the user requests a region change.
What are the options for architecting something like this? Has anyone already done something like this with Datomic Cloud?
I’m providing a global service and I want to reduce latency.
Could you share more detail on what problem you are trying to solve. I understand that you believe that cross region replication is your best answer, but I want to better understand how you arrived at that conclusion. Where are you reducing latency to and from? Is there a measured latency problem I can better understand?
I’m trying to address 2 problems:
1/ latency. I’ve tested using a Sydney based app from Europe and the delay is very noticable. I’m considering AWS Global Accelerator for this as a MVP but, for best UX and global redundancy, running > 1 region will be best
2/ data residency/sovereignty laws. this is a moving target and I want to have options on the table as things change. AU and US have pretty poor legislation on this c.f. European requirements. Having a data center in each region will give me options when dealing with this.
I’m not sure that replication is the only answer. I have a multi-tenant application. I could geographically shard based on the tenant id.
In that scenario, ideally I would also be able to migrate a tenant from 1 shard to another. I guess this is still replication but it would not be frequent.
That context helps!
This could be for all data or just on-demand when the user requests a region change.
So the implication here is that a user can only exist in one region at a time? Is that true and do you care if their data is synced in previous regions? I think that if this is the case, you don’t necessarily need full replication but can create ETL jobs to move customers across regions as needed. This gives you the most flexibility for moving user data bi-directionally. Given that you are concerned about data residency and sovereignty laws this approach also makes more sense so that you do not end up with 1 user across multiple regions unnecessarily.
As a side note, I’d be interested in exploring your latency test further and getting specific details, because I have a sneaking suspicion that with enough data we might find a bottle neck there like caching that could greatly improve performance. We do offer Datomic consulting services in the from of Architecture reviews and I think reviewing your needs would be an interesting exercise. If you’d like I could arrange an introductory call where we can learn more about your system.
Cheers,
Jaret
Thanks Jaret, this confirms my suspicion. In my system, the tenancy is org level instead of user level but doesn’t affect this discussion.
An ETL task does look like the way forward here. One stumbling block is the lack of excision when it comes to data residency. I’m hoping for news from Cognitect on this front so I can keep that data in Datomic.
Re latency: I’m pretty confident in the comparison. It’s like using localhost when in Sydney and hitting the AU region. From Europe the delay is noticable although bearable. I can do some extra things like gzip, aws accelerate etc so I’ll stick with that for now.
I don’t yet have enough revenue to support a Cognitect review but, as soon as I do, I will reach out for an engagement. Thanks.