Excision is very slow for huge number of entities

Datomic Pro
15
  • Environment : Ubuntu 20.01, Datomic on-prem 1.0.6269
  • History :
  1. Start clear datomic: ./bin/transactor -Xms20g -Xmx20g /opt/datomic/config/transactor.properties
  2. Restore DB: ./bin/datomic -Xmx1g -Xms1g restore-db “file:” "datomic:dev://localhost:4334/testExc?password=datomic"
  3. Restart transactor
  4. Run the next script in REPL:
(require '[datomic.api :as d]) 
(def conn (d/connect "datomic:dev://localhost:4334/testExc?password=datomic")) 

(def get-deleted
		(d/q '[:find [?e ...]
		:in $ $hdb
		:where
		[$hdb ?e :IModel/uid ]
		(not [?e :IModel/uid])
		]
		(d/db conn) (d/history (d/db conn)))
		)

(defn makeExciseSt
  [e]
  {:db/excise e})

(def exciseSt (map makeExciseSt get-deleted))

(def trResult (d/transact conn exciseSt))
(def trResultData (.get user/trResult))
(def dbAfter (:db-after user/trResultData))
(def afterBasisT (.basisT dbAfter))
# The question: Does sync-excise  wait on finishing index job which is run in result of excision
(.get (d/sync-excise conn afterBasisT))
(d/gc-storage conn (java.util.Date.))
  1. Restart all peers(including REPL too)
  • Expectation : All entities which were in get-deleted do not existing anymore in datomic
  • Actual : Some number of entities from get-deleted are still available in history DB.
  • Evidence : I’m still able to query entities from get-deleted in history DB
  • Impact : Can’t decrease the size of db to make peers consume less RAM