Stuck with connecting to Datomic Ions during "Getting Started"

Datomic Cloud
1

I had datomic working a few months back. I deleted the cloud formation templates. I reinstalled the system, now I’m getting stuck when following along here: https://docs.datomic.com/cloud/getting-started/connecting.html.

My connection through the bastion servers works as tested by: curl -x socks5h://localhost:$DATOMIC_SOCKS_PORT http://entry.$DATOMIC_SYSTEM.$DATOMIC_REGION.datomic.net:8182/

This line gives warnings which i’m told I can ignore: (def client (d/client cfg))

WARNING: An illegal reflective access operation has occurred …

The next step, (d/create-database client {:db-name "movies"}) doesn’t work.

ExceptionInfo Datomic Client Exception clojure.core/ex-info (core.clj:4739)

I’m not sure how to proceed debugging this. Any tips would be greatly appreciated.

2

So I tried deleting and reinstalling the system multiple times but i kept ending up in this bricked state. I left the broken system installed, and went ahead and installed a second datomic solo instance(once in the same region, and a second in another region). ALL systems are working after having done this (inclduing the one that was previously bricked).

3

Can you provide the full exception stack trace? You can find it in your Cloudwatch logs, under the log stream named “datomic-(your-system-name}”.
https://docs.datomic.com/cloud/operation/monitoring.html#searching-cloudwatch-logs

4

Glad you got it working.
My suspicion is that your AWS account was missing one or more global resources used by Datomic (i.e. IAM roles, etc) that are shared by multiple Datomic systems. Installing a fresh system (with a new name) recreated the missing resources, which is what allowed all the systems to function.

5

I did a filter for Error. It seems there were 3 cloudwatch logs for each call (going off the timestamps).

Log 1

{ "Msg": "Exception writing to storage for db: 9d94802f-efb3-4965-a69c-af84acd9ce73", "Ex": { "Cause": "Alias arn:aws:kms:eu-central-1:301612442803:alias/datomic is not found. (Service: AWSKMS; Status Code: 400; Error Code: NotFoundException; Request ID: 2bc074dd-9c7b-47a8-8213-6db86ce8bae3)", "Via": [ { "Type": "com.amazonaws.services.kms.model.NotFoundException", "Message": "Alias arn:aws:kms:eu-central-1:301612442803:alias/datomic is not found. (Service: AWSKMS; Status Code: 400; Error Code: NotFoundException; Request ID: 2bc074dd-9c7b-47a8-8213-6db86ce8bae3)", "At": [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "handleErrorResponse", "AmazonHttpClient.java", 1640 ] } ], "Trace": [ [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "handleErrorResponse", "AmazonHttpClient.java", 1640 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "executeOneRequest", "AmazonHttpClient.java", 1304 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "executeHelper", "AmazonHttpClient.java", 1058 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "doExecute", "AmazonHttpClient.java", 743 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "executeWithTimer", "AmazonHttpClient.java", 717 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "execute", "AmazonHttpClient.java", 699 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "access$500", "AmazonHttpClient.java", 667 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl", "execute", "AmazonHttpClient.java", 649 ], [ "com.amazonaws.http.AmazonHttpClient", "execute", "AmazonHttpClient.java", 513 ], [ "com.amazonaws.services.kms.AWSKMSClient", "doInvoke", "AWSKMSClient.java", 3202 ], [ "com.amazonaws.services.kms.AWSKMSClient", "invoke", "AWSKMSClient.java", 3178 ], [ "com.amazonaws.services.kms.AWSKMSClient", "executeGenerateDataKey", "AWSKMSClient.java", 1552 ], [ "com.amazonaws.services.kms.AWSKMSClient", "generateDataKey", "AWSKMSClient.java", 1527 ], [ "datomic.kms$generate_data_key", "invokeStatic", "kms.clj", 68 ], [ "datomic.kms$generate_data_key", "invoke", "kms.clj", 68 ], [ "datomic.log$get_secret_key_for_batch", "invokeStatic", "log.clj", 41 ], [ "datomic.log$get_secret_key_for_batch", "invoke", "log.clj", 29 ], [ "datomic.log$write_batch", "invokeStatic", "log.clj", 128 ], [ "datomic.log$write_batch", "invoke", "log.clj", 119 ], [ "datomic.log$init", "invokeStatic", "log.clj", 162 ], [ "datomic.log$init", "invoke", "log.clj", 156 ], [ "datomic.catalog2.Client$init_log__28312", "invoke", "catalog2.clj", 301 ], [ "datomic.catalog2.Client$fn__28316", "invoke", "catalog2.clj", 313 ], [ "clojure.core.async$thread_call$fn__5739", "invoke", "async.clj", 442 ], [ "clojure.lang.AFn", "run", "AFn.java", 22 ], [ "java.util.concurrent.ThreadPoolExecutor", "runWorker", "ThreadPoolExecutor.java", 1149 ], [ "java.util.concurrent.ThreadPoolExecutor$Worker", "run", "ThreadPoolExecutor.java", 624 ], [ "java.lang.Thread", "run", "Thread.java", 748 ] ] }, "Type": "Alert", "Tid": 73, "Timestamp": 1547305056377 }

Log 2

{ "CognitectAnomaliesCategory": "CognitectAnomaliesFault", "DatomicAnomaliesError": { "Error": "Unable to create catalog entry", "Cause": { "Error": "Error initializing log", "Cause": { "Error": "Unable to initialize log", "Cause": { "Error": { "Cause": "Alias arn:aws:kms:eu-central-1:301612442803:alias/datomic is not found. (Service: AWSKMS; Status Code: 400; Error Code: NotFoundException; Request ID: 2bc074dd-9c7b-47a8-8213-6db86ce8bae3)", "Via": [ { "Type": "com.amazonaws.services.kms.model.NotFoundException", "Message": "Alias arn:aws:kms:eu-central-1:301612442803:alias/datomic is not found. (Service: AWSKMS; Status Code: 400; Error Code: NotFoundException; Request ID: 2bc074dd-9c7b-47a8-8213-6db86ce8bae3)", "At": [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "handleErrorResponse", "AmazonHttpClient.java", 1640 ] } ], "Trace": [ [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "handleErrorResponse", "AmazonHttpClient.java", 1640 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "executeOneRequest", "AmazonHttpClient.java", 1304 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "executeHelper", "AmazonHttpClient.java", 1058 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "doExecute", "AmazonHttpClient.java", 743 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "executeWithTimer", "AmazonHttpClient.java", 717 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "execute", "AmazonHttpClient.java", 699 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "access$500", "AmazonHttpClient.java", 667 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl", "execute", "AmazonHttpClient.java", 649 ], [ "com.amazonaws.http.AmazonHttpClient", "execute", "AmazonHttpClient.java", 513 ], [ "com.amazonaws.services.kms.AWSKMSClient", "doInvoke", "AWSKMSClient.java", 3202 ], [ "com.amazonaws.services.kms.AWSKMSClient", "invoke", "AWSKMSClient.java", 3178 ], [ "com.amazonaws.services.kms.AWSKMSClient", "executeGenerateDataKey", "AWSKMSClient.java", 1552 ], [ "com.amazonaws.services.kms.AWSKMSClient", "generateDataKey", "AWSKMSClient.java", 1527 ], [ "datomic.kms$generate_data_key", "invokeStatic", "kms.clj", 68 ], [ "datomic.kms$generate_data_key", "invoke", "kms.clj", 68 ], [ "datomic.log$get_secret_key_for_batch", "invokeStatic", "log.clj", 41 ], [ "datomic.log$get_secret_key_for_batch", "invoke", "log.clj", 29 ], [ "datomic.log$write_batch", "invokeStatic", "log.clj", 128 ], [ "datomic.log$write_batch", "invoke", "log.clj", 119 ], [ "datomic.log$init", "invokeStatic", "log.clj", 162 ], [ "datomic.log$init", "invoke", "log.clj", 156 ], [ "datomic.catalog2.Client$init_log__28312", "invoke", "catalog2.clj", 301 ], [ "datomic.catalog2.Client$fn__28316", "invoke", "catalog2.clj", 313 ], [ "clojure.core.async$thread_call$fn__5739", "invoke", "async.clj", 442 ], [ "clojure.lang.AFn", "run", "AFn.java", 22 ], [ "java.util.concurrent.ThreadPoolExecutor", "runWorker", "ThreadPoolExecutor.java", 1149 ], [ "java.util.concurrent.ThreadPoolExecutor$Worker", "run", "ThreadPoolExecutor.java", 624 ], [ "java.lang.Thread", "run", "Thread.java", 748 ] ] } } } } }, "Msg": "Server Error", "Type": "Alert", "Tid": 72, "Timestamp": 1547305056451 }

Log 3

{ "Msg": "Uncaught Exception: Alias arn:aws:kms:eu-central-1:301612442803:alias/datomic is not found. (Service: AWSKMS; Status Code: 400; Error Code: NotFoundException; Request ID: de69063b-f70a-4807-9dbf-efa8bde77867)", "Ex": { "Cause": "Alias arn:aws:kms:eu-central-1:301612442803:alias/datomic is not found. (Service: AWSKMS; Status Code: 400; Error Code: NotFoundException; Request ID: de69063b-f70a-4807-9dbf-efa8bde77867)", "Via": [ { "Type": "com.amazonaws.services.kms.model.NotFoundException", "Message": "Alias arn:aws:kms:eu-central-1:301612442803:alias/datomic is not found. (Service: AWSKMS; Status Code: 400; Error Code: NotFoundException; Request ID: de69063b-f70a-4807-9dbf-efa8bde77867)", "At": [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "handleErrorResponse", "AmazonHttpClient.java", 1640 ] } ], "Trace": [ [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "handleErrorResponse", "AmazonHttpClient.java", 1640 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "executeOneRequest", "AmazonHttpClient.java", 1304 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "executeHelper", "AmazonHttpClient.java", 1058 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "doExecute", "AmazonHttpClient.java", 743 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "executeWithTimer", "AmazonHttpClient.java", 717 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "execute", "AmazonHttpClient.java", 699 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutor", "access$500", "AmazonHttpClient.java", 667 ], [ "com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl", "execute", "AmazonHttpClient.java", 649 ], [ "com.amazonaws.http.AmazonHttpClient", "execute", "AmazonHttpClient.java", 513 ], [ "com.amazonaws.services.kms.AWSKMSClient", "doInvoke", "AWSKMSClient.java", 3202 ], [ "com.amazonaws.services.kms.AWSKMSClient", "invoke", "AWSKMSClient.java", 3178 ], [ "com.amazonaws.services.kms.AWSKMSClient", "executeGenerateDataKey", "AWSKMSClient.java", 1552 ], [ "com.amazonaws.services.kms.AWSKMSClient", "generateDataKey", "AWSKMSClient.java", 1527 ], [ "datomic.kms$generate_data_key", "invokeStatic", "kms.clj", 68 ], [ "datomic.kms$generate_data_key", "invoke", "kms.clj", 68 ], [ "datomic.index_io$create_index_job_writer", "invokeStatic", "index_io.clj", 59 ], [ "datomic.index_io$create_index_job_writer", "invoke", "index_io.clj", 54 ], [ "datomic.index$create_index$fn__17823$state_machine__5502__auto____17828$fn__17830", "invoke", "index.clj", 795 ], [ "datomic.index$create_index$fn__17823$state_machine__5502__auto____17828", "invoke", "index.clj", 793 ], [ "clojure.core.async.impl.ioc_macros$run_state_machine", "invokeStatic", "ioc_macros.clj", 973 ], [ "clojure.core.async.impl.ioc_macros$run_state_machine", "invoke", "ioc_macros.clj", 972 ], [ "clojure.core.async.impl.ioc_macros$run_state_machine_wrapped", "invokeStatic", "ioc_macros.clj", 977 ], [ "clojure.core.async.impl.ioc_macros$run_state_machine_wrapped", "invoke", "ioc_macros.clj", 975 ], [ "datomic.index$create_index$fn__17823", "invoke", "index.clj", 793 ], [ "clojure.lang.AFn", "run", "AFn.java", 22 ], [ "java.util.concurrent.ThreadPoolExecutor", "runWorker", "ThreadPoolExecutor.java", 1149 ], [ "java.util.concurrent.ThreadPoolExecutor$Worker", "run", "ThreadPoolExecutor.java", 624 ], [ "java.lang.Thread", "run", "Thread.java", 748 ] ] }, "Type": "Alert", "Tid": 19, "Timestamp": 1547305056461 }
7

As I suspected:

"Cause": "Alias arn:aws:kms:eu-central-1:301612442803:alias/datomic is not found.

This indicates that a KMS entry (likely the KMS master key used by Datomic for encrypting data) was deleted at some point. If you had previously used the same system name and manually deleted some parts of it when you were done (i.e. the KMS entries), it’s possible that when you re-created the system those resources were not created.

However, when you created a new system with a different name, the absence of the resources was detected and they were recreated.

8

Instea of deleting the system I should have just set the auto scaling group to 0. Maybe a “warning” in https://docs.datomic.com/cloud/operation/deleting.html might help others avoid this.

9

You can definitely just turn down the ASG.

You can also safely delete the compute stack - what I suspect is that something else was manually deleted (i.e. someone went into the console and found a resource tagged ‘datomic’ and deleted it).

10

In particular, see: Deleting a System