# Ion deployment failure

**URL:** <https://forum.datomic.com/t/ion-deployment-failure/683>\
**Category:** Troubleshooting\
**Created:** [October 30, 2018, 9:25pm UTC](https://forum.datomic.com/t/ion-deployment-failure/683 "2018-10-30T21:25:06Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jarrod](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/jarrod/32/174_2.png) [@jarrod](https://forum.datomic.com/u/jarrod)\
**Post date:** [October 30, 2018, 9:25pm UTC](https://forum.datomic.com/t/ion-deployment-failure/683/1 "2018-10-30T21:25:06Z")

</div>

I’m deploying an ion and it is failing. Where is the first place to start looking for the _smoking gun_? Cloudwatch, step functions, code deploy, or other?

 ![image](https://us1.discourse-cdn.com/flex016/uploads/cognitect/original/1X/bf23b6d4e98a9c9ec8777348962024f29b4b3358.png)

Digging further into the failed script just shows:

```
[stdout]Received 503
[stdout]Received 503
[stdout]Received 503
[stdout]Received 503
[stdout]Received 503
[stdout]Received 503
[stdout]Received 503
[stdout]Received 503
[stdout]Received 503
[stdout]Received 503
[stdout]Received 503
[stdout]Received 503
```

---

<div class="post-metadata">

**Author:** ![jaret](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/jaret/32/45_2.png) [@jaret](https://forum.datomic.com/u/jaret)\
**Post date:** [November 1, 2018, 6:43pm UTC](https://forum.datomic.com/t/ion-deployment-failure/683/2 "2018-11-01T18:43:38Z")

</div>

Hi @jarrod, What is the status of your deploy when you’re seeing this? Are you running solo/production and what CFT version? Generally, I recommend that you start by reviewing CloudWatch (for alerts and messages), but this looks symptomatic of a timeout in loading your deps or your app at the “validateService” step. I’d be curious what changed between deployments that worked and this one.

FYI

Ion monitoring docs:  
[https://docs.datomic.com/cloud/ions/ions-monitoring.html#local-workflow](https://docs.datomic.com/cloud/ions/ions-monitoring.html#local-workflow)

Ion troubleshooting docs with known errors:  
[https://docs.datomic.com/cloud/troubleshooting.html#troubleshooting-ions](https://docs.datomic.com/cloud/troubleshooting.html#troubleshooting-ions)

---

<div class="post-metadata">

**Author:** ![jarrod](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/jarrod/32/174_2.png) [@jarrod](https://forum.datomic.com/u/jarrod)\
**Post date:** [November 5, 2018, 7:27pm UTC](https://forum.datomic.com/t/ion-deployment-failure/683/3 "2018-11-05T19:27:47Z")

</div>

Thanks for the references @jaret, I have looked over those and found them helpful, though not for my particular issue. I did find that it was a dependency that was causing the issue - I was hoping for a more explicit log or error that would indicate which dependency.

I am not quite sure where to find the CFT version. I am running a solo topology.

---

<div class="post-metadata">

**Author:** ![marshall](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/marshall/32/48_2.png) [@marshall](https://forum.datomic.com/u/marshall)\
**Post date:** [November 8, 2018, 3:45pm UTC](https://forum.datomic.com/t/ion-deployment-failure/683/4 "2018-11-08T15:45:36Z")

</div>

Jarrod,

The CFT Version can be found in the “Outputs” tab of the [CloudFormation stack console](https://console.aws.amazon.com/cloudformation/).

As far as identifying dependency issues - when you run the Ion `push` operation you should see an output like:

```auto
{:rev "8baf1c47e0bb62faf68c76cf7fefa05635f2ed01",
 :uname "mt-ion-test",
 :deploy-groups (mt-test-solo),
 :dependency-conflicts
 {:deps
  {commons-codec/commons-codec #:mvn{:version "1.10"},
   com.cognitect/http-client #:mvn{:version "0.1.80"},
   org.slf4j/slf4j-api #:mvn{:version "1.7.14"},
   org.clojure/core.async #:mvn{:version "0.3.442"}},
  :doc
  "The :push operation overrode these dependencies to match versions already running in Datomic Cloud. To test locally, add these explicit deps to your deps.edn."},

```

The first step I would take would be to test your Ion locally with any reported deps from that response explicitly included in your local `deps.edn` file.

-Marshall

---

<div class="post-metadata">

**Author:** ![sekao](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/sekao/32/180_2.png) [@sekao](https://forum.datomic.com/u/sekao)\
**Post date:** [May 26, 2019, 5:18pm UTC](https://forum.datomic.com/t/ion-deployment-failure/683/5 "2019-05-26T17:18:37Z")

</div>

I have a similar problem, except the error code is ScriptTimedOut. The problem began when I changed my compute stack from solo to production. I’ve tried all day long and it keeps timing out at the ValidateService step. The logs just show “[stdout]Received 000” which doesn’t mean much to me.

Update: it actually happens now when I use the solo compute stack. So this may be related to the updated templates and/or updated ion dependencies, because other than that no code has been changed.

---

<div class="post-metadata">

**Author:** ![sekao](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/sekao/32/180_2.png) [@sekao](https://forum.datomic.com/u/sekao)\
**Post date:** [May 27, 2019, 1:53am UTC](https://forum.datomic.com/t/ion-deployment-failure/683/6 "2019-05-27T01:53:38Z")

</div>

I narrowed it down to one problematic dependency: leiningen. I was using leiningen as a library, and for some reason it didn’t like that. Luckily I was using it for a pretty narrow purpose so I was able to remove it entirely, and now the deploys work for me. I basically had to use the process of elimination to figure it out. Definitely would have been nice to see the cause the of timeout, but I guess AWS doesn’t provide that.

---

<div class="post-metadata">

**Author:** ![ckws](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.datomic.com/ckws/32/303_2.png) [@ckws](https://forum.datomic.com/u/ckws)\
**Post date:** [May 28, 2019, 1:13pm UTC](https://forum.datomic.com/t/ion-deployment-failure/683/7 "2019-05-28T13:13:09Z")

</div>

I ran into the same problem.

For me, it was a missing lib (Apache Commons IO).

I looked in the CloudWatch logs for `datomic-<system>`, based on post [Where to look in AWS when deploy (not push) fails?](https://forum.datomic.com/t/where-to-look-in-aws-when-deploy-not-push-fails/729), and found compiler exception because of a missing library:

```json
{
    "Msg": "LoadIonsFailed",
    "Ex": {
        "Via": [
            {
                "Type": "clojure.lang.Compiler$CompilerException",
                "Message": "Syntax error compiling at (ring/middleware/multipart_params.clj:1:1).",
                "Data": {
                    "ClojureErrorPhase": "CompileSyntaxCheck",
                    "ClojureErrorLine": 1,
                    "ClojureErrorColumn": 1,
                    "ClojureErrorSource": "ring/middleware/multipart_params.clj"
                },
                "At": [
                    "clojure.lang.Compiler",
                    "load",
                    "Compiler.java",
                    7647
                ]
            },
            {
                "Type": "java.lang.ClassNotFoundException",
                "Message": "org.apache.commons.io.IOUtils",
                "At": [
                    "java.net.URLClassLoader",
                    "findClass",
                    "URLClassLoader.java",
                    382
                ]
            }
        ],

...

```
