Today my cloudwatch log is full of messages that says RestartingDaemonException
"Message": "Unable to load index root ref <uuid>", with two different uuid’s. It started this morning, and happens about 3 times/minute.
Do you know what could be the cause of that, and how can I repair it?
Max,
Can you restart your compute node(s) and let us know if the messages continue afterward?
I’ve restarted and the message did go away. I got several ClientSPIAnomaly messages on startup though:
{
"Msg":"ClientSPIAnomaly",
"DatomicClientSpiErrorResponse":{
"Status":200,
"Body":{
"CognitectAnomaliesCategory":"CognitectAnomaliesUnavailable",
"CognitectAnomaliesMessage":"Loading database"
}
},
"Type":"Event",
"Tid":98,
"Timestamp":1571298528204
}
Also, random errors make me nervous and I’d like to know why a restart was required.
I have recently deleted databases, if that’s related.
Thank you for getting back!
This morning the errors are back again 
Hi Max,
The errors you have described are usually transient, and if they do not correlate with user-visible issues are not a problem.
If you post the full stack trace for a RestartingDaemonException I can confirm this for a specific instance.
If transient it seems to linger for long. It’s still happening three times per minute per uuid.
This is the full log entry of the latest instance:
{
"Msg": "RestartingDaemonException",
"Ex": {
"Via": [
{
"Type": "clojure.lang.ExceptionInfo",
"Message": "Unable to load index root ref 37cada6d-4d4b-4faa-8671-4d4f3ae0fce0",
"Data": {
"Ret": {},
"DbId": "37cada6d-4d4b-4faa-8671-4d4f3ae0fce0"
},
"At": [
"datomic.index$require_ref_map",
"invokeStatic",
"index.clj",
843
]
}
],
"Trace": [
[
"datomic.index$require_ref_map",
"invokeStatic",
"index.clj",
843
],
[
"datomic.index$require_ref_map",
"invoke",
"index.clj",
836
],
[
"datomic.index$require_root_id",
"invokeStatic",
"index.clj",
849
],
[
"datomic.index$require_root_id",
"invoke",
"index.clj",
846
],
[
"datomic.adopter$start_adopter_thread$fn__21647",
"invoke",
"adopter.clj",
67
],
[
"datomic.async$restarting_daemon$fn__10442$fn__10443",
"invoke",
"async.clj",
162
],
[
"datomic.async$restarting_daemon$fn__10442",
"invoke",
"async.clj",
161
],
[
"clojure.core$binding_conveyor_fn$fn__5739",
"invoke",
"core.clj",
2030
],
[
"datomic.async$daemon$fn__10439",
"invoke",
"async.clj",
146
],
[
"clojure.lang.AFn",
"run",
"AFn.java",
22
],
[
"java.lang.Thread",
"run",
"Thread.java",
748
]
],
"Cause": "Unable to load index root ref 37cada6d-4d4b-4faa-8671-4d4f3ae0fce0",
"Data": {
"Ret": {},
"DbId": "37cada6d-4d4b-4faa-8671-4d4f3ae0fce0"
}
},
"Type": "Alert",
"Tid": 198,
"Timestamp": 1571733277389
}
Hi Max,
How many different UUIDs are there? This normally can happen transiently after a database has been deleted. Have you/do you delete databases regularly (e.g, as part of a test suite?)
Stu
