Data corruption after restoring a database

Hi, I noticed very strange data contained in database after restoring DB, what I did:

  1. Create a database backup using the command from a server1: ./bin/datomic -Xmx1g -Xms1g backup-db “datomic:dev://localhost:4334/dev-oms?password=datomic” "file:/tmp/"
  2. Restore a database on a server 2: /opt/datomic/bin/datomic -Xmx1g -Xms1g restore-db “file:/tmp/dev-oms” "datomic:dev://localhost:4334/dev-oms?password=datomic"
  3. Add new attributes(:BusinessActionDefinition/* and :ParamDefinition/*) to the restored db and some new entities with these attributes
  4. Remove the database: (d/delete-database “datomic:dev://localhost:4334/dev-oms?password=datomic”)
  5. Repeat restoring by executing step 2

After the last step I see something very strange data:


According to the above image there are 4 entities which refers this one through :ParamDefinition/description, but actually :ParamDefinition/description is string with carnality one, :ParamDefinition/name is string too, but the value stored here is keyword. Vice versa situation for :BusinessActionDefinition/refType which type is keyword but we are seeing here string value:

P.S.

  1. Actually mentioned attributes should not exist in restored database.
  2. Actually :ParamDefinition/* and :BusinessActionDefinition/* can’t be used by single entity
  3. Names of mentioned attributes are aliases

Datomic version: 1.06222 On-Prem

Actually mentioned attributes should not exist in restored database

Were these attributes present in the backup DB? Why should they not be present in the restored DB? Backup/restore does not remove retracted attributes or anything from the origin DB for that matter. I am not following your example here completely. Are you saying you restored, then freshly added new attributes to the DB, then deleted the DB and restored again?

Can you explain why are you performing a backup and restore?

Additionally, you can look at the output from list-backups to see all the the approximate points in time (t) of different available backups:

bin/datomic list-backups backup-uri

Please note that Datomic Backup/Restore is not intended as tool for “forking” a DB, but you can restore into a URI that already points to a different point-in-time for the same database. You cannot restore into a URI that points to a different database.

Hi @jaret.

Were these attributes present in the backup DB?

No, these attributes were created by me after restoring the database, so I mean that after second restoring(before which I did (d/delete-database)) I do not expect to see these attributes again. The fourth step is done on server 2

Can you explain why are you performing a backup and restore?

As a developer sometimes I get a database backup from dev/qa/prod environment to restore this backup on clear datomic server which is on my local machine, then I do some actions(create attributes, new data) on that database restored on y local machine then I delete( datomic.api - Datomic Clojure API documentation)**) the database. After some time I might need to use this backup again and after the second restoring I see data which I was expected was previously deleted by (d/delete-database)

You cannot restore into a URI that points to a different database.

No db is present when I’m restoring a database(unless (d/delete-database) does not delete a database), so I can’t get which different URL you mean

but you can restore into a URI that already points to a different point-in-time for the same database.

In my case the database to be restored is absent on the datomic server, so I don’t need to care about point-in-times

I would ask in another way: can a data from a database deleted by the command (d/delete-database) somehow seen again?

If I do the next steps:

  1. Create database DB1
  2. Put some data D1
  3. Create a backup BCKP1 of DB1
  4. Delete database DB1
  5. Restore BCKP1 to DB1
  6. Put some data D2
  7. Delete database DB1
  8. Restore BCKP1 to DB1

Is it correct that after executing step 8 I see data D2 created at step 6?

P.S. Restoring is executed by “bin/datomic restore-db”

Hi @nikolayandr,

Can you clarify if you are restarting the peers and transactors after a restore. The complete instructions include that you should run restore with all processes down, but for storages of dev the storage resides within the transactor process.

:dev and :free storages currently require a running transactor during restore, because storage resides inside the transactor process. You must start the transactor before running a restore, and then shutdown and restart the transactor after the restore completes.

https://docs.datomic.com/on-prem/backup.html#restoring

Hi, thank you, it helps, but could you say whether it is possible somehow to avoid restarting of transactor since another DBs are served on the same transactor too and we would like keep it available to be able to send transactions for another DBs

Another question, should peers which use other databases on the same transactor be restarted too?

Hi, thank you, it helps, but could you say whether it is possible somehow to avoid restarting of transactor since another DBs are served on the same transactor too and we would like keep it available to be able to send transactions for another DBs

No. You need to restart the transactor.

Another question, should peers which use other databases on the same transactor be restarted too?

Yes, all peers and transactor processes need to be restarted.

If you want to avoid DB1 restore bringing down DB2 you should serve them from separate transactor systems.

Backup and restore are normally used for migration or disaster recovery. It seems like you may have a more robust use. Could you share what your objective is of using backup/restore in this manner so that we can understand your use case? What is the problem you are trying to solve here?