Unable to load index root ref <uuid>

Today my cloudwatch log is full of messages that says RestartingDaemonException
"Message": "Unable to load index root ref <uuid>", with two different uuid’s. It started this morning, and happens about 3 times/minute.

Do you know what could be the cause of that, and how can I repair it?

Max,

Can you restart your compute node(s) and let us know if the messages continue afterward?

I’ve restarted and the message did go away. I got several ClientSPIAnomaly messages on startup though:

{  
   "Msg":"ClientSPIAnomaly",
   "DatomicClientSpiErrorResponse":{  
      "Status":200,
      "Body":{  
         "CognitectAnomaliesCategory":"CognitectAnomaliesUnavailable",
         "CognitectAnomaliesMessage":"Loading database"
      }
   },
   "Type":"Event",
   "Tid":98,
   "Timestamp":1571298528204
}

Also, random errors make me nervous and I’d like to know why a restart was required.

I have recently deleted databases, if that’s related.

Thank you for getting back!

This morning the errors are back again :frowning:

Hi Max,

The errors you have described are usually transient, and if they do not correlate with user-visible issues are not a problem.

If you post the full stack trace for a RestartingDaemonException I can confirm this for a specific instance.

If transient it seems to linger for long. It’s still happening three times per minute per uuid.

This is the full log entry of the latest instance:

{
    "Msg": "RestartingDaemonException",
    "Ex": {
        "Via": [
            {
                "Type": "clojure.lang.ExceptionInfo",
                "Message": "Unable to load index root ref 37cada6d-4d4b-4faa-8671-4d4f3ae0fce0",
                "Data": {
                    "Ret": {},
                    "DbId": "37cada6d-4d4b-4faa-8671-4d4f3ae0fce0"
                },
                "At": [
                    "datomic.index$require_ref_map",
                    "invokeStatic",
                    "index.clj",
                    843
                ]
            }
        ],
        "Trace": [
            [
                "datomic.index$require_ref_map",
                "invokeStatic",
                "index.clj",
                843
            ],
            [
                "datomic.index$require_ref_map",
                "invoke",
                "index.clj",
                836
            ],
            [
                "datomic.index$require_root_id",
                "invokeStatic",
                "index.clj",
                849
            ],
            [
                "datomic.index$require_root_id",
                "invoke",
                "index.clj",
                846
            ],
            [
                "datomic.adopter$start_adopter_thread$fn__21647",
                "invoke",
                "adopter.clj",
                67
            ],
            [
                "datomic.async$restarting_daemon$fn__10442$fn__10443",
                "invoke",
                "async.clj",
                162
            ],
            [
                "datomic.async$restarting_daemon$fn__10442",
                "invoke",
                "async.clj",
                161
            ],
            [
                "clojure.core$binding_conveyor_fn$fn__5739",
                "invoke",
                "core.clj",
                2030
            ],
            [
                "datomic.async$daemon$fn__10439",
                "invoke",
                "async.clj",
                146
            ],
            [
                "clojure.lang.AFn",
                "run",
                "AFn.java",
                22
            ],
            [
                "java.lang.Thread",
                "run",
                "Thread.java",
                748
            ]
        ],
        "Cause": "Unable to load index root ref 37cada6d-4d4b-4faa-8671-4d4f3ae0fce0",
        "Data": {
            "Ret": {},
            "DbId": "37cada6d-4d4b-4faa-8671-4d4f3ae0fce0"
        }
    },
    "Type": "Alert",
    "Tid": 198,
    "Timestamp": 1571733277389
}

Hi Max,

How many different UUIDs are there? This normally can happen transiently after a database has been deleted. Have you/do you delete databases regularly (e.g, as part of a test suite?)

Stu

Hi!

There are a few, 1-3 uuid’s, and the messages seems to stay for 1-2 days.

If this should be ignored, I would prefer if it wasn’t flagged as an Alert.