Peer server won't use AWS WebIdentity credential provider

Datomic Pro
1

Hi.

We are running Datomic peer server in an AWS EKS cluster. We use Kubernetes service accounts combined with EKS-issued OIDC tokens/Assumable IAM roles to grant access to AWS resources such as DynamoDB and S3.

Unfortunately, even the newest release of datomic-pro is packaged with an outdated version of the AWS Java SDK (1.11.600). Unlike versions after 1.11.704, it does not prioritize WebIdentity over instance profile credentials in the provider chain. This means that even though our peer server service account has the correct IAM roles available, peer server is using the instance role instead. More info can be found in this aws-sdk-java issue.

I’ve solved this by switching out the AWS SDK files in the datomic_pro lib/ directory for newer versions in my container:

RUN rm lib/aws-java-sdk-*.jar
ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk/1.11.704/aws-java-sdk-1.11.704.jar lib/
ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-core/1.11.704/aws-java-sdk-core-1.11.704.jar lib/
ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-dynamodb/1.11.704/aws-java-sdk-dynamodb-1.11.704.jar lib/
ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-sts/1.11.704/aws-java-sdk-sts-1.11.704.jar lib/

However, it’s not a great long term solution. I’m posting this just in case others encounter the same problem. I’m also hoping that someone on the datomic team can tell me if/when the aws sdk will be updated in the datomic-pro release.

1 Like
2

Hi Jonathan,

We’re working on an update to the AWS SDK dependency for an upcoming release.

-Marshall

3

Hello @jonathan ,

I am running datomic on EKS too, and it has been running just fine, but now I want to backup to s3, so I started thinking of using service account and wanted to find out if datomic supports it and stumbled on your post. Thanks. I am on the latest datomic (1.0.6222) and it is still on the 1.11.600 aws sdk.
My question is any reason why you removed about 180 jars and replaced it only with 4 ? Have you not had any issues because of the missing 176 ? So would I just have to add aws-java-sdk-s3 to your list, since I am trying to backup to s3 ?