Best Practices for Managing Schema Migrations in Datomic!

General
1

Hello everyone,

I am relatively new to Datomic and have been exploring its powerful features. One thing that I am unsure about is how to handle schema migrations efficiently. In a traditional relational database…, schema changes are often managed with migration scripts but I am struggling to figure out the best way to do this in Datomic.

In my application…, the schema evolves over time, and I want to make sure that I am following best practices to avoid data corruption or inconsistencies. Specifically, I have a few questions :-

How can I track schema changes over time in Datomic: ??
What are some recommended approaches for updating schema without downtime or data loss: ??
Are there any tools or patterns for safely deploying schema changes in production: ??

I would appreciate any insights or examples from people who have managed schema migrations in a Datomic environment. I have also searched on the forum for the solution related to my query and found this thread https://forum.datomic.com/t/does-schema-migrations-is-an-issue-in-datomic-aws-certification-course but still looking for some more help and tips.

Thanks in advance !!

With Regards
Daniel Jose

2

Hi @danielljose,

Welcome to Datomic!

Datomic supports online schema evolution, meaning you can modify schema while the system is running. No DB downtime to transact schema changes! You can alter schema to

  • rename attributes
  • rename your own programmatic identities (uses of :db/ident)
  • add or remove indexes
  • add or remove uniqueness constraints
  • change attribute cardinality
  • change whether history is retained for an attribute
  • change whether an attribute is treated as a component

Since Datomic is immutable you can easily track schema changes over time. Each schema change is a transaction in Datomic and you can review schema changes as you would any transaction. You submit a transaction with the new facts, including the :db/id and the new value(s) of the attribute(s). All changes happen synchronously. There are only a few invariants to worry about and Datomic will abort the transaction if your proposed schema change violates one of those invariants and you will need to correct the data before transacting that change. An example of this would be attempting to alter a schema attribute to unique while non-unique values exist for that attribute.

see: Changing Schema | Datomic

You can use the sync-schema api to coordinate schema changes and ensure your peer has a db connection with all schema changes through a given t.

Tracking and reviewing schema changes:

(def hist-db (d/history db))

;; Find all schema changes to attributes over time
(d/q '[:find ?attr ?tx ?op
       :in $
       :where
       [?e :db/ident ?attr ?tx ?op]]
     hist-db)
=>{[:artist.type/other 13194139534315 true]
   [:artist.type/misc 13194139534315 false]}

Depending your applications needs you may want to consider maintaining a schema version entity. Something like:

(d/transact conn [{:db/ident :schema/version
                   :db/valueType :db.type/long
                   :db/cardinality :db.cardinality/one}])
(d/transact conn [{:schema/version 2}])

It’s also a great idea to annotate your schema, like:

;; Instead of destructive change, you can
;; 9. Annotate your schema
{:db/ident :user
 :schema/see-instead :user2
 :db/doc "prefer the user2 namespace for new development"}

In fact there is some great Schema growth advice here: day-of-datomic/samples/schema_growth.clj at master · Datomic/day-of-datomic · GitHub.

I’ll be interested if more community folks chime in on this thread as there are a lot of interesting schema change applications I have seen in support and I believe there is a lot of advice I am missing. But I hope this gets you started.

3

Hi! I recently needed to understand schema changes, and these articles were helpful to me: