Frustrating first experience

[dakra]

Hi,

I played around a little bit with the free Datomic OnPrem version and I like the general
concept of Datomic. So I thought I give Datomic Cloud a try, especially with Ions it seems
if I could get everything setup I would be really fast in prototyping my ideas (and even use it in production).
Host re-frame app in S3/Cloudfront, User management with Cognito, API Gateway, Datomic and I just have write
my app logic in Clojure and everything else is done for me. Sound great.

So I start at the beginning with the first Setup “Start a System” page

There (and in the following pages) I already wonder about lots of dead links.
E.g. just on this first page the topology, access gateway, access gateway and CloudFormation failure links are not working.

In the template I have “Solo” and “Solo 2” and only if I select the latest but one I see the cheap price,
so I choose that one and follow the rest of the tutorial.
I get successfully the Dashboard created and my compute and storage instances.
I start datomic-access and curl -x socks ... works and I get a successful s3-auth-path response.
But then when I actually try to create a database I get “Unable to find keyfile at s3://[…]/datomic/access/admin/.keys”.
I asked in clojurians but no help so I try
a bit more as I thought maybe I have the wrong user, so I used (and from now on forward only use) the root user with all access.
Same problem.
So I decide to remove the cloud fromation and start again.
Terminate cloud formation and re-creating didn’t work because it couldn’t create the storage.
Then I tried a different name and same thing.
Then I saw I made a mistake because I didn’t follow https://docs.datomic.com/cloud/operation/deleting.html
So I delete everything this time completely and try again.
Another error and then rollback from the template.
I do this another time until I try creating a system with a different name.
Now I have a running Datomic Cloud instance again and I even can connect and create a Database!

Now I try the “Ions” tutorial. I do everything as described and the deploy fails like
for other people here or here.
I also tried with a very minimalistic “echo” only Ion but this also fails to deploy.

Looging in the logs I also see a lot of RestartingDaemonException (like here).

As you always have to wait when (re-)creating the cloud formation I feel I “wasted” already
over a day just trying to literally follow step by step what’s written in the tutorial
and it simply doesn’t work and it doesn’t really give any meaningful error messages to debug
and I feel kinda lost even though I haven’t even startet yet

Thanks for reading my long rant
-Daniel

[marshall]

Hi,

I’m sorry your first experience has been frustrating.
I’ve fixed the links you mentioned on the setup page.

As far as the keyfile issue you were seeing, that is usually indicative of AWS user permissions. I’m glad you were able to resolve it with a new stack, however.

As far as the Ion deploy issue, we would need to look at the specific errors in your Datomic system logs to determine the issue with the deploy. You can find details on how to search your logs here: Where to look in AWS when deploy (not push) fails? and https://docs.datomic.com/cloud/operation/monitoring.html#searching-cloudwatch-logs

-Marshall

[stu]

Hi Daniel,

Thank you so much for your candid and detailed feedback. You can ignore the RestartingDaemonException – the “restarting” means that Datomic is auto-recovering, so this is only ever a problem if it accelerates into a retry storm.

We are continuing to work on the other issues you reported!

-Stu

[dakra]

Thanks for your answer and fixing the links.

Regarding the Ion deploy issue. I don’t see anything in the logs.
My system is named “arlo” and I simply copy the sample ion-config to resources/datomic/ion-config.edn and change the :app-name to “arlo”.
Then:

[daniel@matrix ion-starter]$ clojure -A:dev -m datomic.ion.dev '{:op :push}'

Downloading: com/datomic/java-io/0.1.11/java-io-0.1.11.pom from s3://datomic-releases-1fc2183a/maven/releases/
(cognitect.s3-libs.s3/upload "datomic-code-72cb23de-ba2d-46e2-be50-f1c904d1b383" [{:local-zip ".datomic-ions/datomic/apps/arlo/stable/a03c59f4
be1af937d72878a7ab2f8cbb08bcb999.zip", :s3-zip "datomic/apps/arlo/stable/a03c59f4be1af937d72878a7ab2f8cbb08bcb999.zip"}] {:op :push})
{:rev "a03c59f4be1af937d72878a7ab2f8cbb08bcb999",
 :deploy-groups (arlo-Compute-1XWVSV8WYVSL4),
 :deploy-command
 "clojure -Adev -m datomic.ion.dev '{:op :deploy, :group arlo-Compute-1XWVSV8WYVSL4, :rev \"a03c59f4be1af937d72878a7ab2f8cbb08bcb999\"}'",
 :doc
 "To deploy to arlo-Compute-1XWVSV8WYVSL4, issue the :deploy-command"}


[daniel@matrix ion-starter]$ clojure -Adev -m datomic.ion.dev '{:op :deploy, :group arlo-Compute-1XWVSV8WYVSL4, :rev "a03c59f4be1af937d72878a7
ab2f8cbb08bcb999"}'

{:execution-arn
 arn:aws:states:eu-central-1:545023107959:execution:datomic-arlo-Compute-1XWVSV8WYVSL4:arlo-Compute-1XWVSV8WYVSL4-a03c59f4be1af937d72878a7ab2f
8cbb08bcb99-1572355546212,
 :status-command
 "clojure -Adev -m datomic.ion.dev '{:op :deploy-status, :execution-arn arn:aws:states:eu-central-1:545023107959:execution:datomic-arlo-Comput
e-1XWVSV8WYVSL4:arlo-Compute-1XWVSV8WYVSL4-a03c59f4be1af937d72878a7ab2f8cbb08bcb99-1572355546212}'",
 :doc
 "To check the status of your deployment, issue the :status-command."}


[daniel@matrix ion-starter]$ clojure -Adev -m datomic.ion.dev '{:op :deploy-status, :execution-arn arn:aws:states:eu-central-1:545023107959:ex
ecution:datomic-arlo-Compute-1XWVSV8WYVSL4:arlo-Compute-1XWVSV8WYVSL4-a03c59f4be1af937d72878a7ab2f8cbb08bcb99-1572355546212}'

{:deploy-status "FAILED", :code-deploy-status "FAILED"}

And looking in the logs I see nothing related:

I think it might be a problem that I created and deleted the Datomic
system (/cloud formation) a couple of times as I wrote in my original post.

I first created a datomic system/app with name “wadoso” and there
I could deploy the Ion tutorial, but then there was the problem with
the database connection (which probably was my fault with wrong
access rights).

Then I couldn’t delete and re-create a datomic system with the same
name and after a few tries I could create one with a different one
(“arlo” in my case) and there I now have the above problem with
Ion deployment.

I don’t use AWS in any other way so if there is some “hard reset”
option for AWS which deletes all my storage/instances/etc I would
do that, but that doesn’t seem to exist.
(I could also grant you temporary access to my AWS account if
that’s somehow possible and makes debugging easier.)

Thank you very much for your help.
-Daniel

[jaret]

Daniel,

(I could also grant you temporary access to my AWS account if
that’s somehow possible and makes debugging easier.)

This would be ideal. I am going to shoot you an e-mail from our support portal to create a ticket where we can share account access instructions.

-Jaret

[jaret]

Daniel and I worked together to review the system. By reviewing the code deploy failure reason in the deployments console of CodeDeploy. We identified that the error for the failed deploy was access denied on the first step of the deployment.

Specifically, we determined that the access error was failing on the datomic-code bucket. Further investigation showed that this was a result of the deleting a shared Datomic System resource (specifically the datomic code bucket) this resulted in a mismatch between a newly created datomic code bucket and a pre-existing IAM role.

The current docs around deletion do not mention deleting an IAM role which is why Daniel’s existing IAM role was pointing to a stale bucket. We are going to address this in the documentation after we discuss with the product team. I will post an update once we have addressed this in the docs.

[dakra]

Thank you for the very fast and professional help.

Can’t wait to do more with Datomic and Ions

[lucasjordan]

I recently had this same problem. Specifically, I created the datomic stack twice (I immediately regretting my naming decision after creating the stack the first time). This resulted in the role:

arn:aws:iam::$(aws-account-id):role/$(system-name)-Compute-1SBCPHWNTFCM2-us-east-1

referring to the first s3 bucket ‘datomic-code-$(random-stuff)’ and not the second one. The first bucket did not exist anymore, so I felt confident to change the policy to point to the new datomic-code bucket.

Specifically the role has a policy named:
datomic-code-us-east-1$(more-random-stuff)

This role has two statements, one for s3:ListBucket and one for s3:Get, both of these statements need to be updated to the current, existing datomic-code bucket. If you look at the s3 console you should see a single bucket with name starting with ‘datomic-code-’, copy paste the name of that bucket into the statements.