We are running Datomic peer server in an AWS EKS cluster. We use Kubernetes service accounts combined with EKS-issued OIDC tokens/Assumable IAM roles to grant access to AWS resources such as DynamoDB and S3.
Unfortunately, even the newest release of datomic-pro is packaged with an outdated version of the AWS Java SDK (1.11.600). Unlike versions after 1.11.704, it does not prioritize WebIdentity over instance profile credentials in the provider chain. This means that even though our peer server service account has the correct IAM roles available, peer server is using the instance role instead. More info can be found in this aws-sdk-java issue.
I’ve solved this by switching out the AWS SDK files in the datomic_pro
lib/ directory for newer versions in my container:
RUN rm lib/aws-java-sdk-*.jar ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk/1.11.704/aws-java-sdk-1.11.704.jar lib/ ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-core/1.11.704/aws-java-sdk-core-1.11.704.jar lib/ ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-dynamodb/1.11.704/aws-java-sdk-dynamodb-1.11.704.jar lib/ ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-sts/1.11.704/aws-java-sdk-sts-1.11.704.jar lib/
However, it’s not a great long term solution. I’m posting this just in case others encounter the same problem. I’m also hoping that someone on the datomic team can tell me if/when the aws sdk will be updated in the datomic-pro release.