# Where to look in AWS when deploy (not push) fails?

**URL:** <https://forum.datomic.com/t/where-to-look-in-aws-when-deploy-not-push-fails/729>\
**Category:** Troubleshooting\
**Created:** [December 10, 2018, 9:26am UTC](https://forum.datomic.com/t/where-to-look-in-aws-when-deploy-not-push-fails/729 "2018-12-10T09:26:14Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![vvvvalvalval](https://avatars.discourse-cdn.com/v4/letter/v/d9b06d/32.png) [@vvvvalvalval](https://forum.datomic.com/u/vvvvalvalval)\
**Post date:** [December 10, 2018, 9:26am UTC](https://forum.datomic.com/t/where-to-look-in-aws-when-deploy-not-push-fails/729/1 "2018-12-10T09:26:14Z")

</div>

So, after pushing then running the deploy-status command for about 2 minutes, I got the dreaded `{:deploy-status "FAILED", :code-deploy-status "FAILED"}`.

Where can I go from there? I’d look in Cloudwatch logs, but there are more than 30 log groups created by the stack, so it’s a bit of a needle in a haystack.

I went to see in CodeDeploy, and all I found there was

```auto
The overall deployment failed because too many individual instances failed deployment, too few healthy instances are available for deployment, or some instances in your deployment group are experiencing problems

```

as well as the event:

```nohighlight
Error code: ScriptFailed

Script name: scripts/deploy-validate

Message: Script at specified location: scripts/deploy-validate run as user datomic failed with exit code 1 

Logs:

[stdout]Received 503
[stdout]Received 503
// [... a few dozens of the same elided]
[stdout]Received 503
[stdout]WARN: validation did not succeed after two minutes

```

This does not seem to be related to the application code, as deploying a revision that previously worked fails similarly.

Generally speaking, a checklist for troubleshooting Ions deployment (or more detailed error messages) would be appreciated.

Cheers,

---

<div class="post-metadata">

**Author:** ![marshall](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/marshall/32/48_2.png) [@marshall](https://forum.datomic.com/u/marshall)\
**Post date:** [December 10, 2018, 5:37pm UTC](https://forum.datomic.com/t/where-to-look-in-aws-when-deploy-not-push-fails/729/2 "2018-12-10T17:37:13Z")

</div>

Hi Val,

You should look at the specific log group for your Datomic system. It is named `datomic-<your-system-name>`

The details of the code failure in your ion deployment should appear in that group.

We intend to improve the clarity of naming of the other (internal) log groups going forward.

---

<div class="post-metadata">

**Author:** ![jzwolak](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/jzwolak/32/450_2.png) [@jzwolak](https://forum.datomic.com/u/jzwolak)\
**Post date:** [November 11, 2020, 7:54pm UTC](https://forum.datomic.com/t/where-to-look-in-aws-when-deploy-not-push-fails/729/3 "2020-11-11T19:54:45Z")

</div>

I believe this answer is out of date. My guess is that the naming of internal groups has been changed and that the deployment logs no longer go in `datomic-<your-system-name>`. I see groups with `CreateCodeDeploy`, `CreateLambdaFromArray`, `GetDeploymentStatus`, and others in the name. I imagine one of those will have it.

I see these in “CloudWatch -\> Logs -\> Log groups”.

If anyone knows more precisely where this information is, I’d appreciate the help because I cannot find it either.

---

<div class="post-metadata">

**Author:** ![jzwolak](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/jzwolak/32/450_2.png) [@jzwolak](https://forum.datomic.com/u/jzwolak)\
**Post date:** [November 11, 2020, 9:10pm UTC](https://forum.datomic.com/t/where-to-look-in-aws-when-deploy-not-push-fails/729/4 "2020-11-11T21:10:39Z")

</div>

I actually found an error message in “CodeDeploy -\> Deployments” and the deployment that was in progress. And it was a strange one, probably good for starting another topic. AWS displayed the error right at the top of the deployment information page “memory allocation error” for the `rm` command! I reset the instances in the deployment autoscaling group and everything works fine.

---

<div class="post-metadata">

**Author:** ![cch1](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/cch1/32/470_2.png) [@cch1](https://forum.datomic.com/u/cch1)\
**Post date:** [July 26, 2023, 5:42pm UTC](https://forum.datomic.com/t/where-to-look-in-aws-when-deploy-not-push-fails/729/5 "2023-07-26T17:42:08Z")

</div>

@jzwolak , we have seen this same problem twice in the past week. Did you ever figure out the root cause and a resolution?

---

<div class="post-metadata">

**Author:** ![cch1](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/cch1/32/470_2.png) [@cch1](https://forum.datomic.com/u/cch1)\
**Post date:** [July 26, 2023, 5:44pm UTC](https://forum.datomic.com/t/where-to-look-in-aws-when-deploy-not-push-fails/729/6 "2023-07-26T17:44:50Z")

</div>

Forgot to mention that this “memory allocation error” only started happening after we updated our Datomic Cloud stacks to the latest (free) version. ← @jaret

---

<div class="post-metadata">

**Author:** ![jzwolak](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/jzwolak/32/450_2.png) [@jzwolak](https://forum.datomic.com/u/jzwolak)\
**Post date:** [July 26, 2023, 5:59pm UTC](https://forum.datomic.com/t/where-to-look-in-aws-when-deploy-not-push-fails/729/7 "2023-07-26T17:59:46Z")

</div>

@cch1 , I’m not sure what you mean by “free”. Datomic changed it’s license in April 2023 such that all Datomic versions are free. The AWS resources however, have never been free for any version of Datomic.

I never figured out what caused the memory allocation problem. I have not seen it since and have deployed many times. Perhaps something was fixed or I upgraded to a more recent version. I’m confident I am not running the latest version at the time of this writing.

Things have been working for me for some time now (including new deployments) without a problem.

---

<div class="post-metadata">

**Author:** ![cch1](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/cch1/32/470_2.png) [@cch1](https://forum.datomic.com/u/cch1)\
**Post date:** [July 26, 2023, 10:24pm UTC](https://forum.datomic.com/t/where-to-look-in-aws-when-deploy-not-push-fails/729/8 "2023-07-26T22:24:02Z")

</div>

I am referring to the Datomic Cloud CF stacks that implement the “free” (of Cognitect charges) version.

---

<div class="post-metadata">

**Author:** ![jaret](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/jaret/32/45_2.png) [@jaret](https://forum.datomic.com/u/jaret)\
**Post date:** [August 1, 2023, 3:04pm UTC](https://forum.datomic.com/t/where-to-look-in-aws-when-deploy-not-push-fails/729/9 "2023-08-01T15:04:04Z")

</div>

> [@cch1](#):
>
> Forgot to mention that this “memory allocation error” only started happening after we updated our Datomic Cloud stacks to the latest (free) version. ← @jaret

Are you still seeing this? Could I review your logs? What size instance are you running?

I’d love to open a case to track this down:

[https://support.cognitect.com/hc/en-us/requests/new](https://support.cognitect.com/hc/en-us/requests/new)

---

<div class="post-metadata">

**Author:** ![cch1](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/cch1/32/470_2.png) [@cch1](https://forum.datomic.com/u/cch1)\
**Post date:** [August 1, 2023, 3:40pm UTC](https://forum.datomic.com/t/where-to-look-in-aws-when-deploy-not-push-fails/729/10 "2023-08-01T15:40:07Z")

</div>

Hi Jaret,  
We upgraded our instance size from the smallest to the next-to-smallest (t3.medium) and that seems to have resolved the issue. Prior to the upgrade we were running our development environment and our production environment on the same sized instances _and only having the problem in production_. Worth noting that that we have no users live yet so the load was minimal in both cases. The one notable difference: I had detailed metrics enabled in prod but not in dev.
