Ion deploy fails due to Access Denied

Datomic Cloud Troubleshooting
1

Following the ion tutorial.

On first deploy, when doing {:op :deploy-status} I get:

{:deploy-status "FAILED", :code-deploy-status "FAILED"}

Looking in AWS CodeDeploy, it states ‘Most recent event’ is DownloadBundle, and when I click on View Events:
50

What role might be missing for the ‘Access Denied’ error at this stage?

2

Hi @Andre, you’ll want to confirm you have sourced the same AWS credentials required to connect to your Datomic Cloud system. Please check that you can connect via proxy/repl as described in tutorial:

https://docs.datomic.com/cloud/getting-started/configuring-access.html
https://docs.datomic.com/cloud/getting-started/connecting.html

I am going to pull all the required permissions. But I wanted to ask, can you run as administrator?

3

Yes I can connect with repl via socks proxy fine, create db, do transactions and queries.

I’ll try as Administrator tomorrow (late here in UK :grinning:).

4

Same issue when using Administrator.

A couple of things that might be of interest:

  • I originally a different system, (never got as far as code deploy, just wanted to see what is involved in get Datomic Cloud system running). I deleted that one, and started over - trying to deploy to new system
  • I have not gone through the ‘first upgrade’ process - is this essential and maybe the cause for this? I am only running Solo typology.
5

I started fresh, with a brand new AWS account, and everything works fine when I follow the tutorial.

I successfully deployed using both the ‘root’ AWS account, and an IAM account with AdministratorAccess and datomic-admin-xxxx-eu-west-1 roles attached.

So unfortunately I still don’t know what was wrong with previous setup, but I’m very happy that I can now proceed.

6

Hi Andre,

I’m glad you got the issue resolved.

The problem you encountered suggests that somehow the Datomic instance in your system was missing a permission required to read the Ion package from S3.

Is it possible that you were running an older version of Datomic Cloud (from before the release of Ions)?

-Marshall

7

Andre,

I realize this is an older topic, but I wanted to update it in case anyone finds it when searching.

The steps to check/repair the issue are as follows:

  • Go to EC2 instances
  • Select a compute instance
  • Click IAM role
  • Open Policy name
  • Open Datomic Code Policy
  • Take note of the “Resource” key’s "arn:aws:s3:::datomic-code value
  • Go to S3 in another tab
  • Does the datomic-code bucket there match the value in the Code Policy?
  • If not, go back to IAM policies page
  • Edit Policy
  • Go to the Json tab
  • Replace BOTH datomic-code values in the json with the actual bucket arn you saw in S3
  • Review
  • Save changes

This circumstance appears to occur when a user inadvertently deletes the datomic-code bucket from S3. We are also investigating a situation where the bucket appears to have been removed by AWS without having been deleted by the user.

-Marshall

6 Likes
8

This really helped. Any update on AWS removing the bucket. I still can’t figure out why this happened since I created a new system from scratch.

9

The most common cause of this is manually renaming/deleting/altering resources managed by Datomic.
The datomic-code bucket is shared between all Datomic systems in your AWS account (per region), so the issue is not tied to creation/deletion of an individual system, but rather the “global” bucket used by all systems.

10

Just wanted to say thank you for writing this here! I hit this exact issue and had it fixed in a couple of minutes thanks to your comment.

I have no idea what caused this particular issue for us. As far as I know, no one tampered with the Datomic bucket (no DeleteBucket events in CloudTrail).

1 Like