REPL on the Compute notes?

[eneroth]

We had an issue this morning, same symptoms as described over in…

API Gateway internal server error

This kind of looks like a cold start issue, since it only happens after I make a request against API gateway routes after not using them for a little while. The first request always returns an internal server error to me. From the associated lambda logs, I see this error: java.io.IOException: Connection reset by peer: datomic.ion.lambda.handler.exceptions.Fault datomic.ion.lambda.handler.exceptions.Fault: java.io.IOException: Connection reset by peer at datomic.ion.lambda.handler$throw_anomaly…

… except this time it was persistent. I solved it by terminating the node. It returned 500 on requests, but for some reason the NLB was perfectly fine with this and didn’t terminate it automatically.

We’re now looking at more liberal sprinkling of casts throughout the code to catch it next time when/if it happens, but this seems a bit like black box engineering. It would have been nice to have a more direct way to inspect the state of the machine.