Failed Production Storage update to 884-9095

Datomic Cloud Troubleshooting
1

I successfully updated a Solo stack but when I moved to the Production Storage stack, CloudFormation tells me that

EnsureAdminPolicyLogGroup
Replacement type updates not supported on stack with disable-rollback.

then fails the Storage update.
I noticed in CloudFormation under Stack failure options the option to Roll back all stack resources. I haven’t tried it for fear of breaking the storage of my primary database (and the doc does not recommend changing any options), though it seems like that option may get the CloudFormation stack update going again.

Would love to hear recommendations!

2

Hi @adamtait, What version are you upgrading from? Are you currently saying that the “roll back all stack resources” option is selected?

3

On version - my DatomicCFTVersion output in my CF stack is 616 and I think I last upgraded in 2020 so (probably) my version is https://docs.datomic.com/cloud/changes.html#616-8879

In the couple of failed upgrade attempts, I’ve left the Stack failure options to the default Preserve successfully provisioned resources.

Thanks @jaret!

4

@jaret -
any thoughts on how I might get the upgrade to succeed?
do think switching the Stack failure options to Roll back all stack resources is risky? Do you think it would even help?

5

@adamtait Do you have some time to pair together? I believe the only issue preventing Datomic from upgrading in this instance is either admin privileges/privilege related or a modified resource (as Datomic will not delete any resource modified since deployment). I think it might be best if we work together in real time to confirm. Apologies for losing the thread on this.

6

Circling back here we resolved this issue and upgraded the stacks. The initial issue was that the first stack upgrade needed to be performed to move from the Marketplace “master and nested stack.” This is discussed in our split stacks doc page:

https://docs.datomic.com/cloud/operation/split-stacks.html

1 Like
7

I posted the same thing on Slack, but this place seems to be suited better:

I also just tried to upgrade an existing datomic production system from 715-8973 to 884-9095 Datomic Cloud Change Log | Datomic and got this error while upgrading the storage:

UPDATE_FAILED The following resource(s) failed to update: [EnsureAdminPolicyLogGroup].

My stack has a root stack with the name of my system and two childs: compute & storage. It was also created via the AWS Marketplace. I just found this ticket and I’m wondering if I should also split our stack and try upgrading again?

Thanks in advance!

8

I worked with @Xceno. Cross posting here to close the loop in the event that someone finds this thread and wants to know how we resolved.

There were a few resources which Cloud Formation would not delete (because they had been modified). The items we deleted were a VPC LINK, associated EIPs, VPC Endpoint, Internet Gateways, and VPC endpoints. We also had to clear a few route tables. Unfortunately these items don’t have a determined order for deletion and are tied to AWS. Once deleted the master stacks finished deleting and he was able to re-launch using our split-stack instructions.

1 Like