Ions push/deployments automation issues

Hi guys,

Trying to automate ions push/deployment using CodeBuild, I get stuck with an S3 error:

{:deploy-status ERROR, :message VPC endpoints do not support cross-region requests (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied)}

To build, I use the clojure:tools-deps Docker image with the clojure -Adev tools/release.clj command that runs the function in this gist I’m using as a base as saw in the Ions `datomic.ion.dev` documentation? post.

When running the Docker container locally on my machine with my AWS credentials it works fine, but not on CodeBuild.

I tried to set up CodeBuild to use my Datomic cluster’s VPC, without luck.
I made sure that CodeBuild has the permissions to access the Datomic code S3 bucket.
Also I’m on the same region for everything (Datomic, CodeBuild…) which is eu-west-1 (Ireland), hence my surprise on the cross-region error.

I also tried doing the same thing from a Jenkins instance we have on EC2 for other projects and got the same issue.

Am I missing something here?

CodeBuild uses a VPC endpoint to access S3. VPC endpoints are region-scoped for bucket access.

Because of this, the ‘instance’ that CodeBuild runs can only download from an S3 bucket in the same region where it is running.

The Datomic Maven repo (which is specified in your deps.edn) that hosts the ion-dev (and other) artifacts is hosted via an S3 bucket in us-east-1.

What this means is that you will need to run CodeBuild in us-east-1 until we are able to provide a cross-region S3 path for access to the Datomic maven repository.
Alternatively, you could host your own internal private Maven repo in your region of choice and specify it as the repository address in your deps.edn file, but then you’d be responsible for keeping it up to date when new versions of packages are released.

Thanks for your response!

I thought it wasn’t an issue linked to the Datomic Maven repo as the error occurs after all the dependencies download logs.

Nevertheless, I tried your suggestion and added the dependencies in the S3 bucket we use as Maven repo (which is on the same region eu-west-1). All dependencies seems to be downloaded just fine, but I still get the same issue with the VPC endpoint…

{:command-failed "{:op :push :region \"eu-west-1\"}",
 :causes
 ({:message
   "VPC endpoints do not support cross-region requests (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied)",
   :class AmazonS3Exception})}

I also checked that the Jenkins instance (I’m trying this solution before adding CodeBuild into the play) have the permission to write in our Datomic code bucket and I cannot think of another S3 bucket involved in the push operation that could cause the issue.

We have the same issue. From the stacktrace, it doesn’t seem related to Clojure or Datomic.

I made the following steps to a minimum reproducible case:

  1. Create a bucket in a region with default settings (I used us-east-1)
  2. Put a test file in it.
  3. Create a second bucket in a different region with default settings (e.g. eu-west-1)
  4. Run the following code. Note, it is not a datomic app.
(import [com.amazonaws.services.s3  AmazonS3ClientBuilder]
        [com.amazonaws.services.s3.model CopyObjectRequest])

(defn -main [& args]
  (let [client (AmazonS3ClientBuilder/defaultClient)
        request (CopyObjectRequest.
                  "codebuild-test-bucket-us-east-1" ; source bucket
                  "test.txt"                        ; source key
                  "codebuild-test-bucket-eu-west-1" ; dest bucket
                  "test.txt")]                      ; dest key
    (.copyObject client request)))

These are the deps:

{:deps {org.clojure/clojure {:mvn/version "1.9.0"}
        com.amazonaws/aws-java-sdk-core {:mvn/version "1.11.349"}
        com.amazonaws/aws-java-sdk-s3 {:mvn/version "1.11.210"}}}

Locally, everything works fine. Running the CodeBuild container locally also works. Running it in AWS fails with the same error.

Things I’ve tried:

  • Put both buckets in the same region. This works. Copying is successful.
  • Upgrade aws-java-sdk-(core|s3) to the latest version. Still fails.
  • Run the aws s3 cp <source> <dest> command. This also fails.

I’ve not tried using the newer official aws-sdk-java-v2. Perhaps the problem is not present there.

There is no problem downloading code from buckets outside of your region in CodeBuild, this request will go over the internet.
The ion push command seems to directly copy objects from the datomic-releases s3 repo to your datomic-code bucket, which fails if this is cross-region. I.e, for our system in eu-west-1:

  • aws s3 cp <bucket-in-us-east-1> . works. Request goes over the internet.
  • aws s3 cp <local-file> <bucket-in-eu-west-1> works. Request stays within the AWS network
  • aws s3 cp <bucket-in-us-east-1> <bucket-in-eu-west-1> is a cross-region request. This fails.

Our workaround has been to create a VPC with a NAT gateway and a private subnet for CodeBuild to run in. This works for us now, although it would be nice if can be solved. Maybe with s3 cross-region replication so we can use VPC s3 endpoints.