Having a few Datomic Cloud Solo stacks, here’s a sample of what we found updating our code to the new Datomic Cloud (production) stack.
TL;DR we’re really happy with the simplification, unification, flexibility and price of this new topology.
We started by creating a new stack (not updating one of our existing stacks first) to test out our app in a pristine environment.
One surprise we found that was not sufficiently covered in the release notes is the disappearance of the lambda proxy support in favor of Http Direct. We’re very happy with the change and how easy it’s become to use this mode, but it caught us by surprise. With that said, adjusting required simple changes:
So in essence, we’re now able to entirely retire our req->ring utility. This is great news since every clojure shop out there using Ions with lambda proxies had to pump up their own code to massage requests to make them look more ring-like for their HTTP handlers.
Apart from this, some notes:
The new automation for API Gateway works like a charm for our purposes. It’s great to not have to bother setting this up anymore.
The new endpoint URLs backed by API Gateway work well, with the exception that if you map them to a custom domain, e.g. my-db.example.com, you won’t be able to use this in your config maps. This is probably due to the client API’s parsing the endpoint URL to extract e.g. the ID of the system.
The new DB config maps now accept a creds-profile for the AWS configured profile. We appreciate the ease here as each one of our stacks live in a distinct AWS account.
Now that this new stack works and our code is ready to use it, I’m prepping to upgrade our existing stacks. I expect that I’ll only need to delete our API Gateway custom configs before upgrading, to ensure that no conflicts appear during the upgrade.
One final note is that we’re big fans of using remote REPLs. We started with replion. This method worked fine with a Solo topology but will now require adjustments. This is obviously unsupported by Cognitect. I might report back here about this later.
Thanks for the detailed report! I apologize for the missing instruction on moving from lambda proxies to HTTP direct, we will get that fixed.
What did your req->ring utility look like? I was not aware of this issue.
While we do not support remote REPL-ing into Datomic Cloud, we are always keen to hear what you are doing (or struggling to do) so that we can prioritize future enhancements. Thanks again!
(defn- get-path
"Extracts request URI, removing the leading `/<stage>` path added by AWS.
This implementation fails if no such leading stage is found in the path."
[req]
(let [s (get-in req [:datomic.ion.edn.api-gateway/data :requestContext :stage] "stg")
p (get-in req [:datomic.ion.edn.api-gateway/data :requestContext :http :path] "/")]
(get (str/split p (re-pattern (str "/" s))) 1 "/"))) ; get rid of stage leading path!
(defn- get-req-method
"Extracts request method."
[req]
(-> req
(get-in [:datomic.ion.edn.api-gateway/data :requestContext :http :method] "GET")
str/lower-case
keyword))
;; The following libs threaded the same path before us. We might need to look at their code for more inspiration.
;; https://github.com/mhjort/ring-apigw-lambda-proxy/blob/master/src/ring/middleware/apigw.clj
;; https://github.com/jpb/ring-aws-lambda-adapter/blob/master/src/ring_aws_lambda_adapter/core.clj
;; Also, this other example appeared after our implementation and seems to contain great wisdom.
;; https://github.com/cnuernber/cljs-lambda-gateway-example/blob/master/src/gateway_example/proxy_lambda.clj
;; He seems to have learned some tricks from https://github.com/FieryCod/holy-lambda
(defn- req->ring
"Transforms an ionized API Gateway request into a more ring-like request.
See https://github.com/ring-clojure/ring/blob/master/SPEC.
See https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html.
See https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html."
[req]
(assoc req
:server-port (get-in req [:datomic.ion.edn.api-gateway/data :headers :x-forwarded-port] 443)
;;:server-name (get-in req [:headers "host"] "example.com") ; is already present
:remote-addr (get-in req [:datomic.ion.edn.api-gateway/data :headers :x-forwarded-for] "0.0.0.0")
:uri (get-path req)
;;:query-string (generate-query-string
;; (get-in req [:datomic.ion.edn.api-gateway/data :queryStringParameters] {}))
:scheme (keyword (get-in req [:datomic.ion.edn.api-gateway/data :headers :x-forwarded-proto] "https"))
:request-method (get-req-method req)
:protocol (get-in req [:datomic.ion.edn.api-gateway/data :requestContext :http :protocol] "HTTP/1.1")
;;:ssl-client-cert ; ignored
;;:headers ; are already present
;;:body InputStream
))
As for this, Stu, it was great to inspect the state of the live app instead of having to reproduce such state locally before inspecting it. It was especially useful to have this capability to accelerate the development of req->ring BTW, because it was the only way to have a REPL active in the ion entry point. (Since the local dev environment doesn’t visit the ion entry point.) With that said, you could answer “Yes but you could have copied a req map in a scratch file and develop your stuff in your REPL, it’s just data transformation after all!” and you’d be right. We’re learning our RDD habits.
I’m done upgrading our stacks. It was very easy upgrading in place and everything went according to plan.
My next step is to upgrade our dev JVM from 8 to 11 for dev-prod parity purposes (I’m fighting my way through our VS Code docker-based development containers) and then I might experiment more with the blessed way of doing local development with a client connection before working to bring back the capacity to REPL into deployed apps. Thanks!
Thanks for giving us access to Java 11’s new APIs.
Thanks for removing the Socks proxy. Our setup now feels so much simpler! To boot either with a local DB or with a real one, it’s now just a matter of flipping a use-dev-local? switch. No need to remember opening up a tunnel before doing that anymore.
I was able to remotely REPL into an instance like we did with the Solo topology. Obviously this makes more sense when in a stack with only 1 instance. The new topology makes it easier since there no more is any bastion to go through, and the Datomic instances are publicly exposed to the Internet. Still, we’ll probably try more often to use the official, blessed way of working.
You look like you’re improving data.json at a rythm were it might make it possible someday for you to be able to substitute it for Jackson. Can’t wait for this day. I wish you success and progress on this side!
People at Cognitect: has Nubank started using Datomic Cloud Ions themselves in some business-critical projects? If so, please chime up in your official communications channels and tell the world. I’m sure it’ll help more clients adopt the Ion model.
I wish you success and progress on this side!