Datomic Ion Connection Refused

Ok enabling and disabling the require for io.pedestal.http seems to be sufficient to trigger the connection refused.

(ns ion.api
  (:require [datomic.ion.lambda.api-gateway :as apigw]
            #_ <-- adding a ; vs not 
            [io.pedestal.http :as http]))

I’m using https://github.com/pedestal/pedestal.ions and https://github.com/pedestal/pedestal.

deps.edn:

{:paths ["src" "resources"]
 :deps  {org.clojure/clojure             {:mvn/version "1.9.0"}
         org.clojure/core.specs.alpha    {:mvn/version "0.1.24"}
         com.datomic/ion                 {:mvn/version "0.9.34"}
         io.pedestal/pedestal.service    {:mvn/version "0.5.5"
                                          :exclusions  [com.cognitect/transit-clj
                                                        org.clojure/tools.analyzer.jvm
                                                        org.clojure/core.async]}
         io.pedestal/pedestal.ions       {:git/url "https://github.com/pedestal/pedestal.ions.git"
                                          :sha     "ac9a0ddc11521c31e4d617dc10cb912f181e7cae"
                                          :tag     "0.1.1"}
         io.pedestal/pedestal.log        {:mvn/version "0.5.5"
                                          :exclusions  [org.slf4j/slf4j-api]}
         javax.servlet/javax.servlet-api {:mvn/version "3.1.0"}

         ;; Suggested by push tool
         commons-codec/commons-codec     {:mvn/version "1.10"}
         org.clojure/tools.analyzer.jvm  {:mvn/version "0.7.0"}
         org.clojure/tools.reader        {:mvn/version "1.0.0-beta4"}
         org.clojure/core.async          {:mvn/version "0.3.442"}
         com.datomic/java-io #:mvn{:version "0.1.13"},
         com.fasterxml.jackson.core/jackson-core #:mvn{:version "2.9.5"},
         com.amazonaws/jmespath-java #:mvn{:version "1.11.349"},
         com.amazonaws/aws-java-sdk-core #:mvn{:version "1.11.349"},
         com.amazonaws/aws-java-sdk-ssm #:mvn{:version "1.11.349"}}

 :mvn/repos {"datomic-cloud" {:url "s3://datomic-releases-1fc2183a/maven/releases"}}

 :aliases
 {:log   {:extra-paths ["config"]
          :extra-deps  {ch.qos.logback/logback-classic {:mvn/version "1.2.3"
                                                        :exclusions  [org.slf4j/slf4j-api]}
                        org.slf4j/slf4j-api            {:mvn/version "1.7.14"}
                        org.slf4j/jul-to-slf4j         {:mvn/version "1.7.14"}
                        org.slf4j/jcl-over-slf4j       {:mvn/version "1.7.14"}
                        org.slf4j/log4j-over-slf4j     {:mvn/version "1.7.14"}}}
  :dev   {:extra-deps  {io.pedestal/pedestal.service-tools {:mvn/version "0.5.5"}
                        io.pedestal/pedestal.jetty         {:mvn/version "0.5.5"}
                        com.datomic/client-cloud           {:mvn/version "0.8.71"}
                        com.datomic/ion-dev                {:mvn/version "0.9.186"
                                                            :exclusions  [org.slf4j/slf4j-nop]}}}
  :jetty {:extra-paths ["dev"]
          :extra-deps  {io.pedestal/pedestal.jetty {:mvn/version "0.5.5"}}}
  :rebel {:extra-deps {com.bhauman/rebel-readline {:mvn/version "0.1.4"}}
          :main-opts  ["-m" "rebel-readline.main"]}}}

Still working through it =)…

The really frustrating part is that I can’t reproduce this at all at the repl.

Don’t know why I didn’t think of this earlier, but I finally got an exception out by doing the require inside a try/catch in the response.

java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/async/ByteArrayFeeder, compiling:(cheshire/factory.clj:53:11)

Now I’m pretty sure I’ve got jackson in the deps.edn already, but I’ll dig into this tomorrow.

So current plan which I’ve been working off of is that my stack is pretty old compared to the current version, and the issue could have to do with one of the deps I’m providing which is being overridden.

My inability to reproduce the import error in the local repl is beyond frustrating, but fair enough.

I’ve updated the stack which was a little more challenging than I was expecting, mostly as it’s hard to tell when faced with this error message:

$  bash datomic-socks-proxy project-datomic-db

Could not connect to the endpoint URL: "https://tagging.eu-central-1.amazonaws.com/"

what the actual problem was…

It appears that my IAM key which I’m using for auth has changed so with some changes to which AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY we appear to be back in business :)…

However now I’ve got a new issue…

$  clojure -Adev -m datomic.ion.dev '{:op :push :uname "app-test"}'

{:command-failed "{:op :push :uname \"app-test\"}",
 :causes
 ({:message
   "com.amazonaws.client.AwsSyncClientParams.getAdvancedConfig()Lcom/amazonaws/client/builder/AdvancedConfig;",
   :class NoSuchMethodError})}

Ok so if you have the no such method error above, it’s probably because you need to update your aws libs in deps.edn.
EG:

com.amazonaws/jmespath-java     {:mvn/version "1.11.349"}
com.amazonaws/aws-java-sdk-core {:mvn/version "1.11.349"}
com.amazonaws/aws-java-sdk-ssm  {:mvn/version "1.11.349"}

to

com.amazonaws/jmespath-java     {:mvn/version "1.11.562"}
com.amazonaws/aws-java-sdk-core {:mvn/version "1.11.562"}
com.amazonaws/aws-java-sdk-ssm  {:mvn/version "1.11.562"}

which is the current latest.

Whoo, we’re at a new error!

Syntax error compiling at (ring/middleware/multipart_params.clj:1:1)

Is the Syntax error seen when you attempt to perform a push ?

Can you provide:

  • the version of Datomic Cloud you are running
  • your current/latest deps.edn

Generally you should be able to replicate the dependency behavior of loading an Ion by locally overriding any dependencies reported in the result of a push with the versions returned and loading all of your user namespaces at the REPL. Have you been able to attempt doing so?

@marshall The error isn’t seen at all when I try a push.

However that might be because I’m trying to do this at the moment:

(defn test-handler
  "Test Handler"
  [{:keys [headers body] :as request}]
  {:status 200
   :body (pr-str headers body request nil
                 (try
                   (require '[io.pedestal.http :as http])
                   (catch Exception e (str "caught exception: " (.getMessage e)))))})

as I did not get any error when I was doing this previously, it just succeeded, I’ve not tested it since I’ve done the stack update.

As to what version of Datomic Cloud I’m running, it should be the current release:

storage:
DatomicCFTVersion	477
-compute:
DatomicCFTVersion	477
DatomicCloudVersion	8741

The datomic storage doesn’t seem to have a DatomicCloudVersion attribute.

deps.edn:

{:paths ["src" "resources"]
 :deps  {org.clojure/clojure             {:mvn/version "1.10.0"}
         org.clojure/core.specs.alpha    {:mvn/version "0.2.44"}
         org.clojure/spec.alpha          {:mvn/version "0.2.176"}
         com.datomic/ion                 {:mvn/version "0.9.34"}
         commons-codec/commons-codec     {:mvn/version "1.10"}
         org.clojure/tools.analyzer.jvm  {:mvn/version "0.7.0"}
         org.clojure/tools.reader        {:mvn/version "1.0.0-beta4"}
         org.clojure/core.async          {:mvn/version "0.3.442"}
         io.pedestal/pedestal.service    {:mvn/version "0.5.5"
                                          :exclusions  [com.cognitect/transit-clj
                                                        org.clojure/tools.analyzer.jvm
                                                        org.clojure/core.async]}
         io.pedestal/pedestal.ions       {:git/url "https://github.com/pedestal/pedestal.ions.git"
                                          :sha     "ac9a0ddc11521c31e4d617dc10cb912f181e7cae"
                                          :tag     "0.1.1"}
         io.pedestal/pedestal.log        {:mvn/version "0.5.5"
                                          :exclusions  [org.slf4j/slf4j-api]}
         javax.servlet/javax.servlet-api {:mvn/version "3.1.0"}

         com.datomic/java-io             {:mvn/version "0.1.14"}
         com.fasterxml.jackson.core/jackson-core  {:mvn/version "2.9.8"}
         com.amazonaws/jmespath-java     {:mvn/version "1.11.479"}
         com.amazonaws/aws-java-sdk-core {:mvn/version "1.11.479"}
         com.amazonaws/aws-java-sdk-ssm  {:mvn/version "1.11.479"}}

 :mvn/repos {"datomic-cloud" {:url "s3://datomic-releases-1fc2183a/maven/releases"}}

 :aliases
 {:log   {:extra-paths ["config"]
          :extra-deps  {ch.qos.logback/logback-classic {:mvn/version "1.2.3"
                                                        :exclusions  [org.slf4j/slf4j-api]}
                        org.slf4j/slf4j-api            {:mvn/version "1.7.14"}
                        org.slf4j/jul-to-slf4j         {:mvn/version "1.7.14"}
                        org.slf4j/jcl-over-slf4j       {:mvn/version "1.7.14"}
                        org.slf4j/log4j-over-slf4j     {:mvn/version "1.7.14"}}}
  :dev   {:extra-deps  {io.pedestal/pedestal.service-tools {:mvn/version "0.5.5"}
                        io.pedestal/pedestal.jetty         {:mvn/version "0.5.5"}
                        com.datomic/client-cloud           {:mvn/version "0.8.71"}
                        com.datomic/ion-dev                {:mvn/version "0.9.229"
                                                            :exclusions  [org.slf4j/slf4j-nop]}}}
  :jetty {:extra-paths ["dev"]
          :extra-deps  {io.pedestal/pedestal.jetty {:mvn/version "0.5.5"}}}
  :rebel {:extra-deps {com.bhauman/rebel-readline {:mvn/version "0.1.4"}}
          :main-opts  ["-m" "rebel-readline.main"]}}}

When and where do you see the syntax error?

It’s in the api response when I call $API_GATEWAY_INVOKE_URL/dev/about after then nil in the test-handler fn:

nil "caught exception: Syntax error compiling at (ring/middleware/multipart_params.clj:1:1)."

The /about is just a way to invoke {proxy+}

There is a transient dependency conflict somewhere with ring/pedestal and the latest Datomic release.
I’ve been able to replicate this issue using the pedestal sample application (https://github.com/pedestal/pedestal-ions-sample) and will continue tracking it down.
I’ll post an update here once I’ve resolved the issue.

1 Like

Sorry, am I reading that correctly? You’ve managed to reproduce it at the repl using the pedestal-ions-sample?

If that’s the case, then I’ll have to give it a go and see if I can debug it from there.

PS: I tried to reproduce the issue in pedestal-ions-sample at my local repl without success, both with clojure -Adev and clojure -Adev:log:jetty calling (require '[io.pedestal.http :as http]) does not produce a syntax error or any other kind of error.

@marshall: Sorry to disturb, but I’m going to be looking over this today, any updates?

Can you try your original project using ion-dev version 0.9.231 ?

Just tried it, doesn’t seem to be any change?

(ns ion.api)

(defn test-handler
  "Test Handler"
  [{:keys [headers body] :as request}]
  {:status 200
   :body (pr-str headers body request nil
                 (try
                   (require '[io.pedestal.http :as http])
                   (catch Exception e (str "caught exception: " (.getMessage e)))))})

At REPL:

(require '[ion.api :as api])

(def req {:server-port 0, :server-name "localhost", :remote-addr "127.0.0.1", :uri "/", :scheme "http", :request-method :get, :headers {}})

(api/test-handler req)

Produces:

{:status 200, :body "{} nil {:server-port 0, :server-name \"localhost\", :remote-addr \"127.0.0.1\", :uri \"/\", :scheme \"http\", :request-method :get, :headers {}} nil nil"}

I’m going to try and deploy it again and see if anything has changed.

PS: Ok that’s good! It’s now giving:

 nil nil

In lambda instead of a syntax error, I’ll have to see if the rest of the Pedestal stuff works tomorrow, it’s a bit late now, but it appears at least for the moment the problem is gone?

I’m not sure if the changes you made fix the reproduction issue as it appears to now work as it should.

Either way, thanks for your help!

The issue only occurs remotely (when deploying to a node), not when running locally.
Did the newer version of ion-dev resolve the remote issue?

-Marshall

Yes, the issue only occurs remotely, the problem at present is gone. I’m assuming it had something to do with the ion-dev dep as the push command is:

clojure -Adev -m datomic.ion.dev '{:op :push :uname "app-test"}'

I’m still actively developing against it, so I’ll mention if I come across anything else.

Thanks for all your help! Much appreciated =)…

One minor puzzle, and this could be just based on how I’ve setup the project.

I have a route at "/" which doesn’t seem to work?

It’s exactly the same definition as here:

(defn home
  [request]
  (r/response "Hello World!"))

(def routes #{["/" :get (conj common-interceptors `home)]
...)

But navigating to / just gives:

{"message":"Missing Authentication Token"}

I’m going through my API Gateway options to see if I’ve screwed something up or there’s an option in the settings I’ve overlooked.

PS: I can’t see anything in the example given that I’ve missed, but I’ve not managed to get it to work.

I even tried adding an ANY method to the / under API Gateway, above /{proxy+}, but no luck.

Not sure why it’s not working, this also appears to work fine in the local repl.

user=> (def req {:server-port 0, :server-name "localhost", :remote-addr "127.0.0.1", :uri "/", :scheme "http", :request-method :get, :headers {}})
user=> (def reqa {:server-port 0, :server-name "localhost", :remote-addr "127.0.0.1", :uri "/about", :scheme "http", :request-method :get, :headers {}})
user=> (require '[ion.api :as api]
   #_=>         '[ion.service :as service])
user=> ((api/handler service/service) reqa)
{:status 200, :headers {"Strict-Transport-Security" "max-age=31536000; includeSubdomains", "X-Frame-Options" "DENY", "X-Content-Type-Options" "nosniff", "X-XSS-Protection" "1; mode=block", "X-Download-Options" "noopen", "X-Permitted-Cross-Domain-Policies" "none", "Content-Security-Policy" "object-src 'none'; script-src 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' https: http:;", "Content-Type" "text/plain"}, :body "Clojure 1.10.0 - served from /about"}
user=> ((api/handler service/service) req)
{:status 200, :headers {"Strict-Transport-Security" "max-age=31536000; includeSubdomains", "X-Frame-Options" "DENY", "X-Content-Type-Options" "nosniff", "X-XSS-Protection" "1; mode=block", "X-Download-Options" "noopen", "X-Permitted-Cross-Domain-Policies" "none", "Content-Security-Policy" "object-src 'none'; script-src 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' https: http:;", "Content-Type" "text/plain"}, :body "Hello World!"}

It’s a minor point for the moment, so I’ll leave it be for now.

This is an API Gateway configuration-specific issue.

As mentioned here: https://stackoverflow.com/questions/52909329/aws-api-gateway-missing-authentication-token the {proxy+} resource does not match the root for API Gateway requests. If you need/want to serve the root of the domain you’ll need to configure a separate API Gateway resource for that.

-Marshall

Ah ok, so it was probably because I didn’t deploy it, just tried it now and it works perfectly!

Thanks again!

There are a lot more pitfalls than I expected doing this, hopefully documenting all this will be helpful to someone in the future =)…

There seems be some aspect of CORS that triggers and breaks on the OPTIONS request method.

I’ve enabled CORS in API Gateway and deployed it. However I still get a:

{"message": "Internal server error"}

I’ve put the curl request at the end.

I’m pretty sure this is happening somewhere inside API Gateway instead of my app, as I can’t see any of these requests appearing in my CloudWatch logs.

I’ve also setup CORS inside my app by doing:

(defn accept-all [origin] true)

(def service {::http/routes routes
              ::http/allowed-origins accept-all
              ::http/resource-path "/public"
              ::http/chain-provider provider/ion-provider})

Where ::http is [io.pedestal.http :as http].

I’ve seen some fixes online:

Curl Request:

curl -v -X OPTIONS -H "Access-Control-Request-Method: GET" -H "Origin: http://example.com" https://<api-gateway-url>.execute-api.eu-central-1.amazonaws.com/dev/
*   Trying 20.185.171.95...
* TCP_NODELAY set
* Connected to <api-gateway-url>.execute-api.eu-central-1.amazonaws.com (20.185.171.95) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.execute-api.eu-central-1.amazonaws.com
*  start date: Oct  8 00:00:00 2018 GMT
*  expire date: Nov  8 12:00:00 2019 GMT
*  subjectAltName: host "<api-gateway-url>.execute-api.eu-central-1.amazonaws.com" matched cert's "*.execute-api.eu-central-1.amazonaws.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fa0da008600)
> OPTIONS /dev/ HTTP/2
> Host: <api-gateway-url>.execute-api.eu-central-1.amazonaws.com
> User-Agent: curl/7.54.0
> Accept: */*
> Access-Control-Request-Method: GET
> Origin: http://example.com
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 500 
< date: Wed, 05 Jun 2019 18:20:18 GMT
< content-type: application/json
< content-length: 36
< x-amzn-requestid: 92a69f2f-87be-11e9-a7d1-4fb4a5c86205
< x-amz-apigw-id: a0YrZGaSFiAFSDg=
< 
* Connection #0 to host <api-gateway-url>.execute-api.eu-central-1.amazonaws.com left intact
{"message": "Internal server error"}
1 Like

So turns out it’s relatively straightforward to sidestep this problem if your interest is deploying a clojurescript app to connect to the ion.

Just serve it from within, word of warning, this isn’t the best way to do it, I’d preferably be sticking my cljs app in say an s3 bucket/cloudfront serving it from there, but for now this will serve.

I might come back in a month or two to improve the situation here.

There is however a few stumbling blocks to this master plan.

  1. Serving index.html with executable js:
    You need your :resource-path set as well as adjusting :content-security-policy-settings.
    In my case that’s:
(ns ion.service
  (:require [ring.util.response :as r]
            [io.pedestal.http :as http]
            [io.pedestal.http.route :as route]
            [io.pedestal.http.body-params :as body-params]
            [io.pedestal.ions :as provider]))

(defn home
  [request]
  (r/content-type
    (r/resource-response "public/index.html")
    "text/html"))

(def common-interceptors [(body-params/body-params) http/json-body])

(def routes #{["/" :get (conj common-interceptors `home)]
              ["/about" :get (conj common-interceptors `about)]})

(defn accept-all [origin] true)

(def service {::http/routes routes
              ::http/allowed-origins accept-all
              ::http/resource-path "/public"
              ::http/secure-headers {:content-security-policy-settings {:script-src "* 'unsafe-inline'"}}
              ::http/chain-provider provider/ion-provider})
  1. In your clojurescript project add cljsbuild configs to stick stuff in public, I’m using figwheel with re-frame, but this should work with whatever you want!
;;; Stick this inside the [:cljsbuild :builds] vector
;;; NOTE: <backend ion project> can be a relative path!

{:id           "iondev"
 :source-paths ["src/cljs"]
 :figwheel     {:on-jsload "frontend-client.core/mount-root"}
 :compiler     {:main                 frontend-client.core
                :output-to            "<backend ion project>/resources/public/js/compiled/app.js"
                :output-dir           "<backend ion project>/resources/public/js/compiled/out"
                :asset-path           "js/compiled/out"
                :source-map-timestamp true
                :preloads             [devtools.preload
                                       day8.re-frame-10x.preload]
                :closure-defines      {"re_frame.trace.trace_enabled_QMARK_" true
                                       "day8.re_frame.tracing.trace_enabled_QMARK_" true}
                :external-config      {:devtools/config {:features-to-install :all}}}}

    {:id           "ionmin"
     :source-paths ["src/cljs"]
     :compiler     {:main            frontend-client.core
                    :output-to       "<backend ion project>/resources/public/js/compiled/app.js"
                    :output-dir      "<backend ion project>/resources/public/js/compiled/min"
                    :optimizations   :advanced
                    :closure-defines {goog.DEBUG false}
                    :pretty-print    false}}

;;; For completeness I add the line below to the `:clean-targets` vector
"<backend ion project>/resources/public/js/compiled/"

So in my case I can now do:

;; Terminal 1: (repl)
clojure -Adev:log:jetty:rebel

;; Terminal 2: (proxy)
bash datomic-socks-proxy <ion stack name>

;; Terminal 3: (deploy)
clojure -Adev -m datomic.ion.dev '{:op :push :uname "app-test"}'
clojure -Adev -m datomic.ion.dev '{:op :deploy, :group <ion stack name>, :uname "app-test"}'
;;; Then copy/paste result of :status-command to see when it's done

;; Terminal 4: (fig) <- This is the terminal sitting in the frontend-client project
;;; dev
lein figwheel iondev
;;; or do the below before using Terminal 3 to deploy the generated compiled code
lein clean
lein cljsbuild once ionmin

There, that’s basically where I’m at present, hopefully I don’t encounter any more issues =)…

Well I’m back >_<…

I’m just getting these now, not sure why…

Cloudwatch Logs

{:cognitect.anomalies/category :cognitect.anomalies/unavailable, :cognitect.anomalies/message "Connection refused", :clojio/throwable :java.net.ConnectException, :clojio/socket-error :connect, :clojio/at 1562877500359, :clojio/remote "10.213.5.116", :clojio/queue :queued-input, :datomic.ion.lambda.handler/retries 0}

bash datomic-socks-proxy

debug1: channel 2: free: direct-tcpip: listening port 8182 for entry.<ion stack name>.eu-west-1.datomic.net port 8182, connect from 127.0.0.1 port 59565 to 127.0.0.1 port 8182, nchannels 3
debug1: Connection to port 8182 forwarding to socks port 0 requested.
debug1: channel 2: new [dynamic-tcpip]
channel 2: open failed: connect failed: Connection refused
debug1: channel 2: free: direct-tcpip: listening port 8182 for entry.<ion stack name>.eu-west-1.datomic.net port 8182, connect from ::1 port 59566 to ::1 port 8182, nchannels 3

Stacktrace

ERROR i.p.http.impl.servlet-interceptor  - {:msg "Dev interceptor caught an exception; Forwarding it as the response.", :line 314}
clojure.lang.ExceptionInfo: clojure.lang.ExceptionInfo in Interceptor :ion.service/datomic-interceptor - Unable to connect to localhost:8182
        at io.pedestal.interceptor.chain$throwable__GT_ex_info.invokeStatic(chain.clj:35)
        at io.pedestal.interceptor.chain$throwable__GT_ex_info.invoke(chain.clj:32)
        at io.pedestal.interceptor.chain$try_f.invokeStatic(chain.clj:57)
        at io.pedestal.interceptor.chain$try_f.invoke(chain.clj:44)
        at io.pedestal.interceptor.chain$process_all_with_binding.invokeStatic(chain.clj:171)
        at io.pedestal.interceptor.chain$process_all_with_binding.invoke(chain.clj:146)
        at io.pedestal.interceptor.chain$process_all$fn__15663.invoke(chain.clj:188)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at io.pedestal.interceptor.chain$process_all.invokeStatic(chain.clj:186)
        at io.pedestal.interceptor.chain$process_all.invoke(chain.clj:182)
        at io.pedestal.interceptor.chain$enter_all.invokeStatic(chain.clj:235)
        at io.pedestal.interceptor.chain$enter_all.invoke(chain.clj:229)
        at io.pedestal.interceptor.chain$execute.invokeStatic(chain.clj:379)
        at io.pedestal.interceptor.chain$execute.invoke(chain.clj:352)
        at io.pedestal.interceptor.chain$execute.invokeStatic(chain.clj:389)
        at io.pedestal.interceptor.chain$execute.invoke(chain.clj:352)
        at io.pedestal.http.impl.servlet_interceptor$interceptor_service_fn$fn__20627.invoke(servlet_interceptor.clj:351)
        at io.pedestal.http.servlet.FnServlet.service(servlet.clj:28)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678)
        at java.lang.Thread.run(Thread.java:745)
Caused by: clojure.lang.ExceptionInfo: Unable to connect to localhost:8182
        at datomic.client.impl.cloud$get_s3_auth_path.invokeStatic(cloud.clj:175)
        at datomic.client.impl.cloud$get_s3_auth_path.invoke(cloud.clj:166)
        at datomic.client.impl.cloud$create_client.invokeStatic(cloud.clj:207)
        at datomic.client.impl.cloud$create_client.invoke(cloud.clj:190)
        at clojure.lang.Var.invoke(Var.java:384)
        at datomic.client.api.impl$dynarun.invokeStatic(impl.clj:19)
        at datomic.client.api.impl$dynarun.invoke(impl.clj:16)
        at datomic.client.api.impl$dynacall.invokeStatic(impl.clj:26)
        at datomic.client.api.impl$dynacall.invoke(impl.clj:23)
        at datomic.client.api.async$client.invokeStatic(async.clj:66)
        at datomic.client.api.async$client.invoke(async.clj:62)
        at datomic.client.api.sync$client.invokeStatic(sync.clj:77)
        at datomic.client.api.sync$client.invoke(sync.clj:75)
        at clojure.lang.Var.invoke(Var.java:384)
        at datomic.client.api.impl$dynarun.invokeStatic(impl.clj:19)
        at datomic.client.api.impl$dynarun.invoke(impl.clj:16)
        at datomic.client.api.impl$dynacall.invokeStatic(impl.clj:26)
        at datomic.client.api.impl$dynacall.invoke(impl.clj:23)
        at datomic.client.api$client.invokeStatic(api.clj:87)
        at datomic.client.api$client.invoke(api.clj:54)
        at datomic.client.api$client.invokeStatic(api.clj:84)
        at datomic.client.api$client.invoke(api.clj:54)
        at ion.service$fn__23175.invokeStatic(service.clj:97)
        at ion.service$fn__23175.invoke(service.clj:95)
        at clojure.lang.AFn.applyToHelper(AFn.java:154)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$memoize$fn__6862.doInvoke(core.clj:6353)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at ion.service$get_connection.invokeStatic(service.clj:108)
        at ion.service$get_connection.invoke(service.clj:104)
        at ion.service$fn__23186.invokeStatic(service.clj:139)
        at ion.service$fn__23186.invoke(service.clj:135)
        at io.pedestal.interceptor.chain$try_f.invokeStatic(chain.clj:54)
        ... 39 common frames omitted

Rather disconcerting as I don’t believe I’ve done anything so I’m not sure why it’s suddenly not working…


Looking at the docs, it seems my version is older than the current version of 480-8772, however I hope that’s not the reason it’s not working. It would be pretty problematic if my live site stops working because my dependencies are “out of date”…


Ok so a quick sanity test of checking the proxy via:

curl -x socks5h://localhost:$DATOMIC_SOCKS_PORT http://entry.$DATOMIC_SYSTEM.$DATOMIC_REGION.datomic.net:8182/
curl: (7) Failed to receive SOCKS5 connect request ack.

Proxy:

debug1: channel 2: new [dynamic-tcpip]
channel 2: open failed: connect failed: Connection refused
debug1: channel 2: free: direct-tcpip: listening port 8182 for entry.<ion stack name>.eu-west-1.datomic.net port 8182, connect from ::1 port 62825 to ::1 port 8182, nchannels 3

Shows that’s also failing… So next step seems to be to look at cloudformation, but that appears to be fine… A pickle to be sure.


And now it just seems to be working again :thinking: