I don’t think I have found the ultimate solution for this problem yet but I have reached a level in which I’m comfortable sharing what I have because I believe it’ll be useful for other people tackling the same problem.
The reason why I doubt this is the ultimate solution is because it has not been battle tested enough for my taste. I haven’t used it in big applications and I haven’t used in production, maintaining it for months or years.
The problem
We are building SPAs, that is, single page applications. Think Google Maps or GMail. When you request the page, you get a relatively small HTML and a huge JavaScript app. This browser app then renders the page and from now on reacts to your interactions, requesting more data from the server whenever needed but never reloading the whole web page.
The reason to develop an application like this is that the user experience ends up being much better. The app feels faster, snappier, more alive. Reloading the whole page, parsing CSS, JavaScript and HTML is slow, but rendering a snippet of HTML is fast. Furthermore, once you have a full app on the client you can start taking advantage of it, performing, for example, validation, storing data than you won’t request again, etc. which saves talking to the server, making the user experience much better.
The problem, though, is that in the initial request you are not sending any content and many web consumers won’t run JavaScript to render your application. I’m talking about search engine bots, snippet generation bots (like the one Facebook, LinkedIn and Twitter use). Even though it seems Google’s bot is executing some JavaScript, it might not be wise to depend on it.
The solution is to run the client side of the application on the server up to the point of waiting for user interaction, generating the HTML that matches that page, and shipping that to the browser. This also help with the fresh page experience as the user will quickly get some content instead of having to wait for a lot of JavaScript to be parsed, compiled and executed (take a look at GMail and how long it takes to load and show you content).
JavaScript, on the server
Running the client JavaScript on the server is often referred to as isomorphic JavaScript, meaning, same form, that is, same code, running on both server and client. There are several server-side (no windows, headless) JavaScript implementations to chose from:
When choosing my approach I was looking for a simple solution, one with the least moving parts to make it easier to deploy and more stable over time. Nashorn was an immediate winner as it ships with Java 8 and it’s well integrated, hiding away secondary processes and inter-process communication (if it’s happening at all, I’m not sure, and this is good).
Nashorn came with two big issues though:
- It’s slow to create new Nashorn instances (this might be true for all JS implementations).
- The documentation is not great.
I think I have overcame both of this issues, so, without further ado, let’s jump in. You can create a new script engine like this:
(.getEngineByName (ScriptEngineManager.) "nashorn")
ScriptEngineManager has many methods to get a script engine, some use the mime type, or the extension, and with those, you may or may not get Nashorn. I prefer to explicitly request Nashorn as it should be available on all Java 8 installations and I don’t believe we can transparently switch JavaScript implementations as they might be too different.
Once you have a script engine, evaluating code is very easy:
(.eval js-engine "var hello = 'world'")
The method eval can also take files, streams, etc. Invoking a JavaScript method is a bit more involved:
(.invokeMethod ^Invocable js-engine
js-object
"method_name"
(object-array [arg1 arg2 arg3])
That will invoke the method method_name in the JavaScript object js-object which you can obtain this way:
(.eval js-engine "object_name")
There’s a lot more to Nashorn but that’s all we are going to use for implementing server-side JavaScript/ClojureScript.
The application
We’ll start from a reagent application created by:
lein new reagent projectx
which you can start by running:
lein figwheel
You can find all the code for this little application in GitHub: https://github.com/carouselapps/isomorphic-clojurescript-projectx. When you visit the app, you’ll briefly see this:
That page, which you can find in handler.clj, is the actual HTML sent to the browser, before the ClojureScript/JavaScript kicks in:
(def home-page
(html
[:html
[:head
[:meta {:charset "utf-8"}]
[:meta {:name "viewport"
:content "width=device-width, initial-scale=1"}]
(include-css (if (env :dev) "css/site.css" "css/site.min.css"))]
[:body
[:div#app
[:h3 "ClojureScript has not been compiled!"]
[:p "please run "
[:b "lein figwheel"]
" in order to start the compiler"]]
(include-js "js/app.js")]]))
Or in actual HTML:
<html>
<head>
<meta charset="utf-8"/>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<link href="css/site.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="app">
<h3>ClojureScript has not been compiled!</h3>
<p>please run <b>lein figwheel</b> in order to start the compiler</p>
</div>
<script src="js/app.js" type="text/javascript"></script>
</body>
</html>
In production, you’ll normally want to show a message about the application being loaded. Here we are going to try to replace it with the actual rendered application.
After seeing that page briefly, ClojureScript gets compiled to JavaScript, served to the browser, executed and it renders the homepage, which looks like this:
This template conveniently ships with two pre-built pages, the home page and the about page. Click in the link to go to the about page and you’ll see its content but no request was sent to the server. All content was shipped before and the rendering happens client side:
If we request that URL, we’ll se the same loading message and then the about page is going to be shown, but there’s a problem. The server doesn’t know that the about page was being requested because the fragment, the bit after the # in the URL, is not sent to the server.
Proper URLs
The reason why a fragment is used that way is because we don’t want to send a request to the server when we click a link and that’s what browsers do when you go from /blah#bleh to /blah#blih. Thankfully HTML 5 comes to the rescue with its history API. You can learn more about it in Dive into HTML5: Manipulating History for Fun & Profit. If you are wondering whether it’s safe to use this feature already, all current browsers support it (except Opera Mini) and IE since version 10:
To move forward with server side rendering of SPAs you need to switch to HTML5 History, which is implemented in ClojureScript by a library called Pushy. While you are at it, I also recommend to switch to an bidirectional routing library like bidi or silk. To make the long story short, you can look at the diff to implement bidi and Pushy in projectx.
Now that the we are using sane URLs, we need to process them on the server side. In the file handler.clj we’ll find the main HTML template, the routes and the app:
(def home-page
(html
[:html
[:head
[:meta {:charset "utf-8"}]
[:meta {:name "viewport"
:content "width=device-width, initial-scale=1"}]
(include-css (if (env :dev) "css/site.css" "css/site.min.css"))]
[:body
[:div#app
[:h3 "ClojureScript has not been compiled!"]
[:p "please run "
[:b "lein figwheel"]
" in order to start the compiler"]]
(include-js "js/app.js")]]))
(defroutes routes
(GET "/" [] home-page)
(resources "/")
(not-found "Not Found"))
(def app
(let [handler (wrap-defaults #'routes site-defaults)]
(if (env :dev) (-> handler wrap-exceptions wrap-reload) handler)))
home-page will stop being a constant as it’ll be a function on the path and while we are at it, let’s rename it to something more appropriate, like render-app:
(defn render-app [path]
(html
[:html
[:head
[:meta {:charset "utf-8"}]
[:meta {:name "viewport"
:content "width=device-width, initial-scale=1"}]
(include-css (if (env :dev) "css/site.css" "css/site.min.css"))]
[:body
[:div#app
[:h3 "ClojureScript has not been compiled!"]
[:p "please run "
[:b "lein figwheel"]
" in order to start the compiler"]]
(include-js "js/app.js")]]))
The reason why it’s taking the path and not the full URL is that the ClojureScript part of this app works with paths instead of URLs and we’ll need them to be consistent. This is due to how Pushy and likely HTML5 History behave.
The routes will now pass the path to render-app:
(defroutes routes
(GET "*" request (render-app (path request)))
(resources "/")
(not-found "Not Found"))
The function that turns the request into a path is similar to ring.util.request/request-url:
(defn- path [request]
(str (:uri request)
(if-let [query (:query-string request)]
(str "?" query))))
When this change is done, you should see no effect in the running application at all. If you want to confirm things are working properly, you could add this to the render-app function:
[:p path]
and you’ll see the path the server sees before the ClojureScript kicks in. You can see the diff for this step in GitHub: https://github.com/carouselapps/isomorphic-clojurescript-projectx/….
The JavaScript engine
Now things get interesting. The render-app method needs to run some JavaScript, so it’ll create the script engine. First, we need to import it (and also require clojure.java.io , which we’ll be using soon):
(ns projectx.handler
(:require ; ...
[clojure.java.io :as io])
(:import [javax.script ScriptEngineManager]))
After creating the engine, we need to define the variable global because Nashorn doesn’t specify it and reagent needs it. Once that’s done, we are ready to load the JavaScript code:
(defn render-app [path]
(let [js-engine (doto (.getEngineByName (ScriptEngineManager.) "nashorn")
(.eval "var global = this")
(.eval (-> "public/js/app.js"
io/resource
io/reader)))]
; ...
It doesn’t yet render anything, but let’s give it a try, let’s see it load the code or… well… fail:
javax.script.ScriptException: ReferenceError: "document" is not defined in <eval> at line number 2
What’s happening here is that app.js is referring document and Nashorn implements JavaScript, but it’s not a browser, it doesn’t have the global, window or document global objects. Let’s look at the offending file:
var CLOSURE_UNCOMPILED_DEFINES = null;
if(typeof goog == "undefined") document.write('<a href="http://js/out/goog/base.js">http://js/out/goog/base.js</a>');
document.write('<a href="http://js/out/cljs_deps.js">http://js/out/cljs_deps.js</a>');
document.write('if (typeof goog != "undefined") { goog.require("projectx.dev"); } else { console.warn("ClojureScript could not load :main, did you forget to specify :asset-path?"); };');
This is a generated JavaScript file that is loaded by our small HTML file. It in turns causes the rest of the JavaScript files to be loaded but the mechanism it uses works in a browser, not in Nashorn. This is where things get hard.
From the project definition, this is how app.js is built:
:cljsbuild {:builds {:app {:source-paths ["src/cljs" "src/cljc"]
:compiler {:output-to "resources/public/js/app.js"
:output-dir "resources/public/js/out"
:asset-path "js/out"
:optimizations :none
:pretty-print true}}}}
It’s built with no optimizations. One of the optimizations, called whitespace, puts all the JavaScript in a single file, so there’s no document trick to load them, but sadly, it will not work in Figwheel.
The solution I came up with, a hack, is to have two builds. One called app which is what I consider the JavaScript app itself and the other one called server-side, which is the one prepared to run on the server:
:cljsbuild {:builds {:app {:source-paths ["src/cljs" "src/cljc"]
:compiler {:output-to "resources/public/js/app.js"
:output-dir "resources/public/js/app"
:asset-path "js/app"
:optimizations :none
:pretty-print true}}
:server-side {:source-paths ["src/cljs" "src/cljc"]
:compiler {:output-to "resources/public/js/server-side.js"
:output-dir "resources/public/js/server-side"
:optimizations :whitespace}}}}
For sanity’s sake, I changed the output of app to go to the directory called app, instead of out. Running Figwheel will auto-compile app, but not server-side; for that, you also need to run lein cljsbuild auto. Now the application loads with no errors.
We also need to properly configure server-side for the dev and uberjar profiles:
:cljsbuild {:builds {:app {:source-paths ["src/cljs" "src/cljc"]
:compiler {:output-to "resources/public/js/app.js"
:output-dir "resources/public/js/app"
:asset-path "js/app"}}
:server-side {:source-paths ["src/cljs" "src/cljc"]
:compiler {:output-to "resources/public/js/server-side.js"
:output-dir "resources/public/js/server-side"
:optimizations :whitespace}}}}
:profiles {:dev {;...
:cljsbuild {:builds {:app {:source-paths ["env/dev/cljs"]
:compiler {:optimizations :none
:source-map true
:pretty-print true
:main "projectx.dev"}}
:server-side {:compiler {:optimizations :whitespace
:source-map "resources/public/js/server-side.js.map"
:pretty-print true}}}}}
:uberjar {;...
:cljsbuild {:jar true
:builds {:app {:source-paths ["env/prod/cljs"]
:compiler {:optimizations :advanced
:pretty-print false}}
:server-side {:compiler {:optimizations :advanced
:pretty-print false}}}}}}
You might have notice that we are not including env/dev/cljs and env/dev/cljs for server-side. That is because those files call projectx.core/init!, which triggers the whole application to start working, which depends on global objects, like window, which are not present in Nashorn.
With this, even the uberjar loads properly and creates JavaScript engines, but so far, we are not doing any server side rendering. That’s the next step. You can see the full diff for this change in GitHub: https://github.com/carouselapps/isomorphic-clojurescript-projectx/….
