Datomic Ion Connection Refused

So this was working originally, and then I was trying to get pedestal working inside the ion, but now it appears to have broken completely.

I’ve been trying to debug this and now even a relatively simple bit of code doesn’t appear to be working.

I’m just trying to run this:

(ns ion.api
  (:require [datomic.ion.lambda.api-gateway :as apigw]))

(defn test-handler
  "Test Handler"
  [{:keys [headers body] :as request}]
  {:status 200
   :body (pr-str headers body request)})

(def app
  "API Gateway web service ion for main app."
  (apigw/ionize test-handler))

and when I test the lambda I get:

{
  "isBase64Encoded": false,
  "statusCode": 500,
  "headers": {},
  "body": "Connection refused"
}

and in cloudwatch:

{:cognitect.anomalies/category :cognitect.anomalies/unavailable, :cognitect.anomalies/message "Connection refused", :clojio/throwable :java.net.ConnectException, :clojio/socket-error :connect, :clojio/at 1559145398145, :clojio/remote "10.213.44.67", :clojio/queue :queued-input, :datomic.ion.lambda.handler/retries 0}

The only thing I can think of is there’s some kind of compile/deploy error, though it appears to deploy successfully, so I’m really uncertain what the issue is.

My next step is cloning the project and starting it again, but I figured I’d see if anyone had any insights.

Ok, this is worrying, now it seems to magically work?

Not sure what I did to fix it, but from my perspective it’s been broken since Monday. It would be good if there’s some root cause that can be picked up here.

Have there been updates to Datomic Cloud that fixed this behind the scenes?

Ok enabling and disabling the require for io.pedestal.http seems to be sufficient to trigger the connection refused.

(ns ion.api
  (:require [datomic.ion.lambda.api-gateway :as apigw]
            #_ <-- adding a ; vs not 
            [io.pedestal.http :as http]))

I’m using https://github.com/pedestal/pedestal.ions and https://github.com/pedestal/pedestal.

deps.edn:

{:paths ["src" "resources"]
 :deps  {org.clojure/clojure             {:mvn/version "1.9.0"}
         org.clojure/core.specs.alpha    {:mvn/version "0.1.24"}
         com.datomic/ion                 {:mvn/version "0.9.34"}
         io.pedestal/pedestal.service    {:mvn/version "0.5.5"
                                          :exclusions  [com.cognitect/transit-clj
                                                        org.clojure/tools.analyzer.jvm
                                                        org.clojure/core.async]}
         io.pedestal/pedestal.ions       {:git/url "https://github.com/pedestal/pedestal.ions.git"
                                          :sha     "ac9a0ddc11521c31e4d617dc10cb912f181e7cae"
                                          :tag     "0.1.1"}
         io.pedestal/pedestal.log        {:mvn/version "0.5.5"
                                          :exclusions  [org.slf4j/slf4j-api]}
         javax.servlet/javax.servlet-api {:mvn/version "3.1.0"}

         ;; Suggested by push tool
         commons-codec/commons-codec     {:mvn/version "1.10"}
         org.clojure/tools.analyzer.jvm  {:mvn/version "0.7.0"}
         org.clojure/tools.reader        {:mvn/version "1.0.0-beta4"}
         org.clojure/core.async          {:mvn/version "0.3.442"}
         com.datomic/java-io #:mvn{:version "0.1.13"},
         com.fasterxml.jackson.core/jackson-core #:mvn{:version "2.9.5"},
         com.amazonaws/jmespath-java #:mvn{:version "1.11.349"},
         com.amazonaws/aws-java-sdk-core #:mvn{:version "1.11.349"},
         com.amazonaws/aws-java-sdk-ssm #:mvn{:version "1.11.349"}}

 :mvn/repos {"datomic-cloud" {:url "s3://datomic-releases-1fc2183a/maven/releases"}}

 :aliases
 {:log   {:extra-paths ["config"]
          :extra-deps  {ch.qos.logback/logback-classic {:mvn/version "1.2.3"
                                                        :exclusions  [org.slf4j/slf4j-api]}
                        org.slf4j/slf4j-api            {:mvn/version "1.7.14"}
                        org.slf4j/jul-to-slf4j         {:mvn/version "1.7.14"}
                        org.slf4j/jcl-over-slf4j       {:mvn/version "1.7.14"}
                        org.slf4j/log4j-over-slf4j     {:mvn/version "1.7.14"}}}
  :dev   {:extra-deps  {io.pedestal/pedestal.service-tools {:mvn/version "0.5.5"}
                        io.pedestal/pedestal.jetty         {:mvn/version "0.5.5"}
                        com.datomic/client-cloud           {:mvn/version "0.8.71"}
                        com.datomic/ion-dev                {:mvn/version "0.9.186"
                                                            :exclusions  [org.slf4j/slf4j-nop]}}}
  :jetty {:extra-paths ["dev"]
          :extra-deps  {io.pedestal/pedestal.jetty {:mvn/version "0.5.5"}}}
  :rebel {:extra-deps {com.bhauman/rebel-readline {:mvn/version "0.1.4"}}
          :main-opts  ["-m" "rebel-readline.main"]}}}

Still working through it =)…

The really frustrating part is that I can’t reproduce this at all at the repl.

Don’t know why I didn’t think of this earlier, but I finally got an exception out by doing the require inside a try/catch in the response.

java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/async/ByteArrayFeeder, compiling:(cheshire/factory.clj:53:11)

Now I’m pretty sure I’ve got jackson in the deps.edn already, but I’ll dig into this tomorrow.

So current plan which I’ve been working off of is that my stack is pretty old compared to the current version, and the issue could have to do with one of the deps I’m providing which is being overridden.

My inability to reproduce the import error in the local repl is beyond frustrating, but fair enough.

I’ve updated the stack which was a little more challenging than I was expecting, mostly as it’s hard to tell when faced with this error message:

$  bash datomic-socks-proxy project-datomic-db

Could not connect to the endpoint URL: "https://tagging.eu-central-1.amazonaws.com/"

what the actual problem was…

It appears that my IAM key which I’m using for auth has changed so with some changes to which AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY we appear to be back in business :)…

However now I’ve got a new issue…

$  clojure -Adev -m datomic.ion.dev '{:op :push :uname "app-test"}'

{:command-failed "{:op :push :uname \"app-test\"}",
 :causes
 ({:message
   "com.amazonaws.client.AwsSyncClientParams.getAdvancedConfig()Lcom/amazonaws/client/builder/AdvancedConfig;",
   :class NoSuchMethodError})}

Ok so if you have the no such method error above, it’s probably because you need to update your aws libs in deps.edn.
EG:

com.amazonaws/jmespath-java     {:mvn/version "1.11.349"}
com.amazonaws/aws-java-sdk-core {:mvn/version "1.11.349"}
com.amazonaws/aws-java-sdk-ssm  {:mvn/version "1.11.349"}

to

com.amazonaws/jmespath-java     {:mvn/version "1.11.562"}
com.amazonaws/aws-java-sdk-core {:mvn/version "1.11.562"}
com.amazonaws/aws-java-sdk-ssm  {:mvn/version "1.11.562"}

which is the current latest.

Whoo, we’re at a new error!

Syntax error compiling at (ring/middleware/multipart_params.clj:1:1)

Is the Syntax error seen when you attempt to perform a push ?

Can you provide:

  • the version of Datomic Cloud you are running
  • your current/latest deps.edn

Generally you should be able to replicate the dependency behavior of loading an Ion by locally overriding any dependencies reported in the result of a push with the versions returned and loading all of your user namespaces at the REPL. Have you been able to attempt doing so?

@marshall The error isn’t seen at all when I try a push.

However that might be because I’m trying to do this at the moment:

(defn test-handler
  "Test Handler"
  [{:keys [headers body] :as request}]
  {:status 200
   :body (pr-str headers body request nil
                 (try
                   (require '[io.pedestal.http :as http])
                   (catch Exception e (str "caught exception: " (.getMessage e)))))})

as I did not get any error when I was doing this previously, it just succeeded, I’ve not tested it since I’ve done the stack update.

As to what version of Datomic Cloud I’m running, it should be the current release:

storage:
DatomicCFTVersion	477
-compute:
DatomicCFTVersion	477
DatomicCloudVersion	8741

The datomic storage doesn’t seem to have a DatomicCloudVersion attribute.

deps.edn:

{:paths ["src" "resources"]
 :deps  {org.clojure/clojure             {:mvn/version "1.10.0"}
         org.clojure/core.specs.alpha    {:mvn/version "0.2.44"}
         org.clojure/spec.alpha          {:mvn/version "0.2.176"}
         com.datomic/ion                 {:mvn/version "0.9.34"}
         commons-codec/commons-codec     {:mvn/version "1.10"}
         org.clojure/tools.analyzer.jvm  {:mvn/version "0.7.0"}
         org.clojure/tools.reader        {:mvn/version "1.0.0-beta4"}
         org.clojure/core.async          {:mvn/version "0.3.442"}
         io.pedestal/pedestal.service    {:mvn/version "0.5.5"
                                          :exclusions  [com.cognitect/transit-clj
                                                        org.clojure/tools.analyzer.jvm
                                                        org.clojure/core.async]}
         io.pedestal/pedestal.ions       {:git/url "https://github.com/pedestal/pedestal.ions.git"
                                          :sha     "ac9a0ddc11521c31e4d617dc10cb912f181e7cae"
                                          :tag     "0.1.1"}
         io.pedestal/pedestal.log        {:mvn/version "0.5.5"
                                          :exclusions  [org.slf4j/slf4j-api]}
         javax.servlet/javax.servlet-api {:mvn/version "3.1.0"}

         com.datomic/java-io             {:mvn/version "0.1.14"}
         com.fasterxml.jackson.core/jackson-core  {:mvn/version "2.9.8"}
         com.amazonaws/jmespath-java     {:mvn/version "1.11.479"}
         com.amazonaws/aws-java-sdk-core {:mvn/version "1.11.479"}
         com.amazonaws/aws-java-sdk-ssm  {:mvn/version "1.11.479"}}

 :mvn/repos {"datomic-cloud" {:url "s3://datomic-releases-1fc2183a/maven/releases"}}

 :aliases
 {:log   {:extra-paths ["config"]
          :extra-deps  {ch.qos.logback/logback-classic {:mvn/version "1.2.3"
                                                        :exclusions  [org.slf4j/slf4j-api]}
                        org.slf4j/slf4j-api            {:mvn/version "1.7.14"}
                        org.slf4j/jul-to-slf4j         {:mvn/version "1.7.14"}
                        org.slf4j/jcl-over-slf4j       {:mvn/version "1.7.14"}
                        org.slf4j/log4j-over-slf4j     {:mvn/version "1.7.14"}}}
  :dev   {:extra-deps  {io.pedestal/pedestal.service-tools {:mvn/version "0.5.5"}
                        io.pedestal/pedestal.jetty         {:mvn/version "0.5.5"}
                        com.datomic/client-cloud           {:mvn/version "0.8.71"}
                        com.datomic/ion-dev                {:mvn/version "0.9.229"
                                                            :exclusions  [org.slf4j/slf4j-nop]}}}
  :jetty {:extra-paths ["dev"]
          :extra-deps  {io.pedestal/pedestal.jetty {:mvn/version "0.5.5"}}}
  :rebel {:extra-deps {com.bhauman/rebel-readline {:mvn/version "0.1.4"}}
          :main-opts  ["-m" "rebel-readline.main"]}}}

When and where do you see the syntax error?

It’s in the api response when I call $API_GATEWAY_INVOKE_URL/dev/about after then nil in the test-handler fn:

nil "caught exception: Syntax error compiling at (ring/middleware/multipart_params.clj:1:1)."

The /about is just a way to invoke {proxy+}

There is a transient dependency conflict somewhere with ring/pedestal and the latest Datomic release.
I’ve been able to replicate this issue using the pedestal sample application (https://github.com/pedestal/pedestal-ions-sample) and will continue tracking it down.
I’ll post an update here once I’ve resolved the issue.

1 Like

Sorry, am I reading that correctly? You’ve managed to reproduce it at the repl using the pedestal-ions-sample?

If that’s the case, then I’ll have to give it a go and see if I can debug it from there.

PS: I tried to reproduce the issue in pedestal-ions-sample at my local repl without success, both with clojure -Adev and clojure -Adev:log:jetty calling (require '[io.pedestal.http :as http]) does not produce a syntax error or any other kind of error.

@marshall: Sorry to disturb, but I’m going to be looking over this today, any updates?

Can you try your original project using ion-dev version 0.9.231 ?

Just tried it, doesn’t seem to be any change?

(ns ion.api)

(defn test-handler
  "Test Handler"
  [{:keys [headers body] :as request}]
  {:status 200
   :body (pr-str headers body request nil
                 (try
                   (require '[io.pedestal.http :as http])
                   (catch Exception e (str "caught exception: " (.getMessage e)))))})

At REPL:

(require '[ion.api :as api])

(def req {:server-port 0, :server-name "localhost", :remote-addr "127.0.0.1", :uri "/", :scheme "http", :request-method :get, :headers {}})

(api/test-handler req)

Produces:

{:status 200, :body "{} nil {:server-port 0, :server-name \"localhost\", :remote-addr \"127.0.0.1\", :uri \"/\", :scheme \"http\", :request-method :get, :headers {}} nil nil"}

I’m going to try and deploy it again and see if anything has changed.

PS: Ok that’s good! It’s now giving:

 nil nil

In lambda instead of a syntax error, I’ll have to see if the rest of the Pedestal stuff works tomorrow, it’s a bit late now, but it appears at least for the moment the problem is gone?

I’m not sure if the changes you made fix the reproduction issue as it appears to now work as it should.

Either way, thanks for your help!

The issue only occurs remotely (when deploying to a node), not when running locally.
Did the newer version of ion-dev resolve the remote issue?

-Marshall

Yes, the issue only occurs remotely, the problem at present is gone. I’m assuming it had something to do with the ion-dev dep as the push command is:

clojure -Adev -m datomic.ion.dev '{:op :push :uname "app-test"}'

I’m still actively developing against it, so I’ll mention if I come across anything else.

Thanks for all your help! Much appreciated =)…

One minor puzzle, and this could be just based on how I’ve setup the project.

I have a route at "/" which doesn’t seem to work?

It’s exactly the same definition as here:

(defn home
  [request]
  (r/response "Hello World!"))

(def routes #{["/" :get (conj common-interceptors `home)]
...)

But navigating to / just gives:

{"message":"Missing Authentication Token"}

I’m going through my API Gateway options to see if I’ve screwed something up or there’s an option in the settings I’ve overlooked.

PS: I can’t see anything in the example given that I’ve missed, but I’ve not managed to get it to work.

I even tried adding an ANY method to the / under API Gateway, above /{proxy+}, but no luck.

Not sure why it’s not working, this also appears to work fine in the local repl.

user=> (def req {:server-port 0, :server-name "localhost", :remote-addr "127.0.0.1", :uri "/", :scheme "http", :request-method :get, :headers {}})
user=> (def reqa {:server-port 0, :server-name "localhost", :remote-addr "127.0.0.1", :uri "/about", :scheme "http", :request-method :get, :headers {}})
user=> (require '[ion.api :as api]
   #_=>         '[ion.service :as service])
user=> ((api/handler service/service) reqa)
{:status 200, :headers {"Strict-Transport-Security" "max-age=31536000; includeSubdomains", "X-Frame-Options" "DENY", "X-Content-Type-Options" "nosniff", "X-XSS-Protection" "1; mode=block", "X-Download-Options" "noopen", "X-Permitted-Cross-Domain-Policies" "none", "Content-Security-Policy" "object-src 'none'; script-src 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' https: http:;", "Content-Type" "text/plain"}, :body "Clojure 1.10.0 - served from /about"}
user=> ((api/handler service/service) req)
{:status 200, :headers {"Strict-Transport-Security" "max-age=31536000; includeSubdomains", "X-Frame-Options" "DENY", "X-Content-Type-Options" "nosniff", "X-XSS-Protection" "1; mode=block", "X-Download-Options" "noopen", "X-Permitted-Cross-Domain-Policies" "none", "Content-Security-Policy" "object-src 'none'; script-src 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' https: http:;", "Content-Type" "text/plain"}, :body "Hello World!"}

It’s a minor point for the moment, so I’ll leave it be for now.

This is an API Gateway configuration-specific issue.

As mentioned here: https://stackoverflow.com/questions/52909329/aws-api-gateway-missing-authentication-token the {proxy+} resource does not match the root for API Gateway requests. If you need/want to serve the root of the domain you’ll need to configure a separate API Gateway resource for that.

-Marshall

Ah ok, so it was probably because I didn’t deploy it, just tried it now and it works perfectly!

Thanks again!

There are a lot more pitfalls than I expected doing this, hopefully documenting all this will be helpful to someone in the future =)…

There seems be some aspect of CORS that triggers and breaks on the OPTIONS request method.

I’ve enabled CORS in API Gateway and deployed it. However I still get a:

{"message": "Internal server error"}

I’ve put the curl request at the end.

I’m pretty sure this is happening somewhere inside API Gateway instead of my app, as I can’t see any of these requests appearing in my CloudWatch logs.

I’ve also setup CORS inside my app by doing:

(defn accept-all [origin] true)

(def service {::http/routes routes
              ::http/allowed-origins accept-all
              ::http/resource-path "/public"
              ::http/chain-provider provider/ion-provider})

Where ::http is [io.pedestal.http :as http].

I’ve seen some fixes online:

Curl Request:

curl -v -X OPTIONS -H "Access-Control-Request-Method: GET" -H "Origin: http://example.com" https://<api-gateway-url>.execute-api.eu-central-1.amazonaws.com/dev/
*   Trying 20.185.171.95...
* TCP_NODELAY set
* Connected to <api-gateway-url>.execute-api.eu-central-1.amazonaws.com (20.185.171.95) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.execute-api.eu-central-1.amazonaws.com
*  start date: Oct  8 00:00:00 2018 GMT
*  expire date: Nov  8 12:00:00 2019 GMT
*  subjectAltName: host "<api-gateway-url>.execute-api.eu-central-1.amazonaws.com" matched cert's "*.execute-api.eu-central-1.amazonaws.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fa0da008600)
> OPTIONS /dev/ HTTP/2
> Host: <api-gateway-url>.execute-api.eu-central-1.amazonaws.com
> User-Agent: curl/7.54.0
> Accept: */*
> Access-Control-Request-Method: GET
> Origin: http://example.com
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 500 
< date: Wed, 05 Jun 2019 18:20:18 GMT
< content-type: application/json
< content-length: 36
< x-amzn-requestid: 92a69f2f-87be-11e9-a7d1-4fb4a5c86205
< x-amz-apigw-id: a0YrZGaSFiAFSDg=
< 
* Connection #0 to host <api-gateway-url>.execute-api.eu-central-1.amazonaws.com left intact
{"message": "Internal server error"}