Pablo Fernandez

Category: Technical

Posts related to programming or other highly technical stuff that can only be of interest to computer geeks.

  • How I found one of the earliest browsers in history

    ,
    , , , , ,

    Yesterday, the web celebrated its 25th birthday and to join in, I have a little story. A couple of years ago I found a NeXTcube. I’m not going to say where it is to avoid vandalism (the computer is publicly accessible under some circumstances without much oversight), but this is the story. Sir Tim Berners-Lee coded the earliest version of the web in his NeXTcube workstation when he was working at CERN, so, I was always interested in this machines, from a historical/playful point of view.

    The cube that was in front of me was more or less abandoned and I asked the owner if I could play with it. He was very reticent but I was more relentless and I got to play with it. He told me that Next computer belonged, at one point, to CERN and that it has not been used since then. I decided to explore it.

    The first interesting thing I found was a file containing a lot of email addresses from people that seemed to work at CERN or be related to CERN in some form or fashion. The owner of the computer decided to be overly professional and deleted the file.

    The second interesting thing I found completely blew my mind. There was a folder called WorldWideWeb and inside it several files called WorldWideWeb_0.1.0.tar, 0.1.1.tar, 0.2.0.tar and so on. Could this be? I opened one by one and indeed they were apps. I started with the oldest and executed them one by one.

    The first one raised an error as it tried to contact cernvax.cern.ch (this Next cube was disconnected) and then it crashed:

    WorldWideWeb_0.1.0

    I kept on going and eventually one started. It was very plain but I knew what it was. I quickly went back to my terminal, open vi, and wrote a small HTML file, which then I passed as a parameter to the little WorldWideWeb_0.2. It worked… it displayed an h1 as a title!

    I was jumping out of my skin. I don’t want to publish the whole picture to avoid releasing private information, but I’m standing, next to the cube, pointing and what could possible be the earliest version of the web browser that still works today, displaying a web site I just coded (it says Hello World):

    WorldWideWeb_0.2

    Then I discovered the browser allowed me to edit the page, directly there, without having to do anything special, and I remembered that Sir Tim Berners-Lee originally designed the web to be read-write, not read-only.

    That was one of the most exciting moments of my life. When I got home I wrote an email to Sir Tim Berners-Lee, telling him of my finding and where he could find that computer, just in case he wanted to get ahold of those binaries (I couldn’t find any source code anywhere on that machine). He never replied, I don’t know if he ever got my email. I bet he gets a lot of it and that he’s a very busy man.

    Update: explained a bit why I don’t want to reveal where this happened.

  • Free-form version 0.2.0 released

    , , , , , , ,

    We are very happy to announce version 0.2.0 of our form building library Free-form. This version includes:

    The Bootstrap 3 support means that you can have whole fields defined as succinctly as:

    [:free-form/field {:type        :email
                   :key         :email
                   :label       "Email"}]]

    Enjoy!

     

  • Prerenderer 0.2.0 released

    , , , , ,

    We are proud to announce the release of version 0.2.0 of our ClojureScript library Prerenderer, a library to do server side pre-rendering of single page applications. In this release, we include:

    The two first items in the changelog came hand in hand and they are the biggest changes to keep in mind if you are upgrading. We are very happy that we no longer need a fork of re-frame and we would like to extend our gratitude to Mike Thompson for working with us on having the appropriate API to make this happen.

    The change in API means that your Prerenderer module now would look something like this:

    (ns projectx.node
  (:require [prerenderer.core :as prerenderer]))

(defn render-and-send [page-path send-to-browser]
  (send-to-browser (render page-path)))

(set! *main-cli-fn* (prerenderer/create render-and-send "ProjectX"))

    instead of:

    (ns projectx.node
  (:require [cljs.nodejs :as nodejs]
            [prerenderer.core :as prerenderer]))

(defn render [req res]
  (let [page-path (.-path (.parse url (.-url (.-query req))))]
    (.send res (render page-path))))

(set! *main-cli-fn* (prerenderer/create render "ProjectX"))

    Enjoy!

  • A browser in your screensaver is a square peg in a round hole

    ,
    , , ,

    This blog post was originally published in Screensaver Ninja‘s blog.

    We were recently challenged by someone who asked what was so special about adding websites into a screensaver. Perhaps, at first, this doesn’t seem like a tough task but after months of challenging work, I can confirm it is. I realized I never shared exactly why yet, so here it is. Putting a browser into your screensaver is like putting a square peg in a round hole.

    Chromium on Mac

    Screensaver Ninja for Mac is already out there. It’s working and it’s robust, but getting there wasn’t without its pains. Initially, we wanted the Mac and Windows versions to have the exact same rendering engine and thus, we went for Chrome’s WebKit, packaged as Chromium Embedded Framework, or CEF for short.

    open-package-contentsChromium, the open source version of Chrome, follows the same structure as Chrome to handle page isolation: by running it in different processes. On Mac, applications are distributed as bundles which you may know as .app files. They are actually directories and you can inspect the contents by right‐clicking or control‐clicking on it and then choosing Show Package Contents. (Warning: if you change anything the app will likely not work anymore.) When you use CEF, you end up with a secondary bundle application inside your application. This is sort‐of supported, but weird and not without issues.

    In Mac OS X, screensavers are dynamic libraries that are loaded by a special screensaver program. When you make a screensaver, you are not in control of the running program, you just have a few entry points to start doing your animation. The Mac OS X screensaver framework doesn’t like it at all when you have a secondary app bundle that you trigger from your library.

    During this research, we found a bunch of issues, many of which were not clear‐cut as solving it for us might break it for other people. We still needed those issues solved so we wrote scripts that would pre-process CEF and solve them for us. Ultimately we ended up dropping CEF; more on that later.

    Swift

    We decided to use Swift for our project. It’s the new way and we prefer higher level languages whenever possible. Swift saves us a bit of pain with memory management, syntax and other things. But we inadvertently caused ourselves quite a bit of pain. The Mac screensaver framework is still an Objective-C application and since we are not building an app, but a library, we need to write and compile Swift with Objective-C binary compatibility. This is not commonly done so it took us quite a bit of bumping our head against a brick wall to figure it out. Furthermore, not being in control of the program loading our library made getting error messages tricky at best.

    Apple’s WebKit

    When we dropped CEF, our alternative was Apple’s WebKit, which was so much easier to integrate and have running, despite the fact that there are two of them; one that works out of the box, but it’s deprecated, called WebView, and a new one that’s not so well supported, called KWWebView. We played a lot with both and both had the same problem: cookies shared with Safari.

    Apple decided that all users of a web view must share cookies with Safari. This was not acceptable for us because we want to have separate independent sessions and in the future we are even planning on having separate cookie jars per site so that you can, for example, be logged in into two different Twitter accounts at the same time. We contacted Apple about it, paying our tech support fee, and the answer was a resounding “can’t be done, we’ll add it to the list of things that we’ll consider in the future”.

    Challenge accepted Mr Apple! We embarked on a quest to achieve this anyway that took us into the dark innards of the Cookie Jar, debugging it at the assembly level to understand its interface and workings:

    debugging-the-cookie-jar

    I have to admit it, that was fun. After understanding Apple’s cookie jar’s implementation, we wrote a test suite that was exercising it all, as far as we know, including bits that we believe are abandoned. After that we wrote our own implementation that stored the cookies separately and used the same test suite to make sure our implementation was equivalent to Apple’s. This code had to be done in a mix of C and Objective-C. Then we used method swizzling to replace Apple’s with our own and ta-da! Cookie separation.

    Windows events

    The Windows version of Screensaver Ninja is of course not done yet, but we’ve already started working on it and we are partly there. One interesting problem that we run into is that Windows doesn’t help you at all with the workflow of a screensaver. It is of paramount importance to us that while the screensaver is running, nobody should be able to interact with those websites. We don’t want any keystrokes, mouse moves, mouse clicks, etc to reach the pages, otherwise it would be a breach of our security approach to dashboards.

    Windows has a long history all the way back to Windows 1.0 that ran as a little program on top of MS DOS. Awww, good old days. Through the decades, ways to code Windows applications have changed radically and thus the way events travel through applications also did. That means that there’s a lot of different ways for an app to get keystrokes, mouse events, etc. Finding them all and plugging all those holes was not trivial and since we are talking about security this required a lot of testing.

    I’m sure that as we go along, we are going to find many more issues like those in the Windows environment, and we are going to solve them and we are going to strive for elegant, stable, robust code.

     

  • jar-copier version 0.3.0 released

    , , ,

    We just released a new version of jar-copier, 0.3.0, that includes:

    • Better reporting of misconfiguration.
    • Thoroughly testing misconfiguration reporting.
    • Added the possibility to manually specify the jars (not java-agents).

    The main change is the last item, which was planned but now it became clear that some people actually wanted it.

  • to-jdbc-uri 0.4.1 released

    , , ,

    We just released a new version of to-jdbc-uri, 0.4.1. A very important change in this one is that we are consolidating all our libraries into the com.carouselapps group ID, so you need to switch from including it like this:

    [to-jdbc-uri "0.3.0"]

    to including it like this:

    [com.carouselapps/to-jdbc-uri "0.4.1"]

    Aside from that, this release supports RedHat OpenShift style of URL that use the schema postgresql:// instead of postgres://. Courtesy of Pradnyesh Sawant.

    Picture by Joe Hall

  • Don’t forget to clear your client side state when logging a user out

    , , , ,

    When a user logs out from our web site, we are used to clearing the session and that’s it. When you are developing a single page application, you are likely to keep a lot of state on the client side, and that should be cleared too.

    For Ninja Tools, that meant going from the traditional re-frame initialize-db:

    (re-frame/register-handler
  :initialize-db
  (fn [_ _]
    (re-frame/dispatch [:get-current-user])
    {:current-route        nil
     :alerts               (sorted-map)
     :current-user         nil
     :log-in-form          {}
     :registration-form    {}
     :reset-password-form  {}
     :change-password-form {}
     :tools                nil
     :used-tools           nil}))

    to having the initial-db in a re-usable value:

    (def initial-db {:current-route        nil
                 :alerts               (sorted-map)
                 :current-user         nil
                 :log-in-form          {}
                 :registration-form    {}
                 :reset-password-form  {}
                 :change-password-form {}
                 :tools                nil
                 :used-tools           nil})

(re-frame/register-handler
  :initialize-db
  (fn [_ _]
    (re-frame/dispatch [:get-current-user])
    initial-db))

    and our logged-out handler to use it instead of modifying the current estate, which meant going from:

    (re-frame/register-handler
  :logged-out
  (fn [db [_]]
    (routing/redirect-to :home)
    (-> db
        (assoc :current-user nil)
        (alerts/add-alert :success "You are now logged out."))))

    to:

    (re-frame/register-handler
  :logged-out
  (fn [db [_]]
    (routing/redirect-to :home)
    (-> db/initial-db
        (alerts/add-alert :success "You are now logged out."))))

    Since we care so much about security, for us, it’s important to go back to initial-db, and if there’s some state that should survive, we’ll pass it on manually. That is, we’ll be doing whitelisting vs blacklisting.

    Something that we haven’t decided is whether we clear the state on log-out, when the user just clicked the log out link, or logged-out, when when the server has cleared the session.

    The advantage of the former is that we clear all the state as soon as possible, the advantage of the later is that should the log out procedure fail for some reason, the app still has the state and it should still be usable, which might be required for re-trying logging out.

    Recapping that, clearing state immediately behaves better when everything goes well, clearing state later behaves better when something goes wrong during the log out process. We can’t comment one being safer that the other, as clearing the session earlier and failing to log out might leave the user under the impression they successfully logged out when it’s not the case.

    Picture by Ken Hawkins.

  • Automatically converting case between SQL and Clojure

    , , , , , ,

    SQL, at least PostgreSQL, likes using snake case for table names, such as user_name, while in Clojure, kebab case is preferred, such as user-name. When you use the library Yesql you are likely to end up with keywords in snake case unless you do some conversion. In our toy project, Ninja Tools, I wanted to perform these conversions automatically.

    To achieve this automatic conversion I wanted to wrap every single function generated by Yesql and do the conversion both ways. This sounded familiar. Dmitri Sotnikov and I came up with a neat trick to do that in Conman, a connection manager for Yesql, that wraps all the Yesql functions binding them to a connection.

    This code wraps around the result for Conman but if you just need to do something similar with plain Yesql I’d recommend looking at Conman’s code. Normally, this is how you would use Conman:

    (ns ninjatools.db.core
  ;...
  )

(defonce ^:dynamic conn (atom nil))

(conman/bind-connection ninjatools.db.core/conn "sql/queries.sql")]

    and this is the code to do the automatic wrapping to convert case style:

    (ns ninjatools.db.core
  ; ...
  )

(defonce ^:dynamic conn (atom nil))


(ns ninjatools.db.core.queries
  (:require [conman.core :as conman]
            [camel-snake-kebab.core :as csk]
            [camel-snake-kebab.extras :as csk-extras]))

(doall
 (for [yesql-query (conman/bind-connection ninjatools.db.core/conn "sql/queries.sql")]
   (intern 'ninjatools.db.core
           (with-meta (:name (meta yesql-query)) (meta yesql-query))
           (fn [& args]
             (let [args (if (< 1 (count args))
                          args
                          (cons (csk-extras/transform-keys csk/->snake_case (first args))
                                (rest args)))]
               (csk-extras/transform-keys csk/->kebab-case
                                          (apply yesql-query args)))))))

(in-ns 'ninjatools.db.core)

    Let me explain what’s going on here. The namespace of the file is ninjatools.db.core. In this namespace we define an atom, conn, to store the connection and then the madness begins.

    Line 7 defines another namespace, one that is used to store the original functions created by Conman and which we are not likely to ever access directly. On line 11 we do exactly that, we invoke Conman, and thus Yesql, so the file with the queries is read and turn into a bunch of functions in the ninjatools.db.core.queries namespace. This functions are also returned as a sequence that we are going to iterate over.

    In line 12 we call intern to essentially define a function in a different namespace, in this case, the one that matches this file. The name of this new function will be the same as the one defined by Yesql thanks to Clojure’s ability to inspect the meta-data of a function, as we can see in line 13. While we are at it, let’s also make the meta-data be same, just in case.

    Since we don’t know how many arguments the function will take, we accept any amount and if there’s more than one, in line 17 we convert the first one from Clojure’s kebab-case to PostgreSQL’s snake_case. The result goest through the reverse process in line 18.

    Very important for the sake of the rest of the file, line 19 takes us back to the natural namespace for this file. Neat trick, isn’t it? Obviously it would be better if this wasn’t required a lot, which is the goal of issue 108, “Callback when defining queries”.

    Any questions?

    Picture by AAB_BAA

  • Isomorphic JavaScript (with ClojureScript) for pre-rendering single-page-applications, part 3

    , , , , ,

    I was not expecting there to be a part 3 to this series and this third part is also going to be quite different to the first two. In parts 1 and 2 I walked you through an exploration of server side pre-rendering with Nashorn. My naive example worked fine with Nashorn but it didn’t survive encountering the real world.

    Nashorn is not a headless browser, it’s a plain JavaScript engine. It doesn’t implement document or window for example, which were easy to workaround, but it also doesn’t implement setTimeout, setInterval or XMLHttpRequest which are much harder to workaround.

    When I started to look for alternatives I focused on NodeJS because I knew if implemented those things I was missing and V8‘s speed is attractive. Also, the fact the ClojureScript has it as a compilation target made me feel it was well supported, a first class citizen.

    At this point someone might interject and exclaim: What about nodyn.io? nodyn.io is an attempt to bring all the NodeJS goodness to Nashorn and I think it’s a great idea. Sadly, on GitHub we can find this notice:

    This project is no longer being actively maintained. If you have interest in taking over the project, please file an issue.

    I’m not sure if the project got far before being abandoned either.

    Implementing the missing bits of Nashorn in Clojure was tempting. It looks like fun and it also looks like something that might be popular amongst Java users and thus good for the Clojure echo system. I exercised some restrain and moved on.

    In the process of experimenting with NodeJS my code quickly took the form of a library and without further ado, let me introduce you to Prerenderer. The ultimate solution for all your server side pre-rendering needs. I’m not going to show you how to use it here because its page go into a lot of detail already.

    My big concern about prerendering is performance and stability. As I showed in part 2, a naive implementation can behave horribly while in production, sometimes taking up to 17 seconds to serve a page. Prerenderer was not developed with a sample project, like I did with Nashorn, but with a project we are working on called Ninja Tools that uses re-frame and AJAX. Before any modifications to it, this was its performance:

    After enabling Prerenderer, this is how it looks like:

    The average response time went up from 51ms to 362ms. This would generally be a very bad thing. The reason for this is explained in Prerenderer’s documentation:

    […] SPAs are never done. Imagine a SPA that has a timer and every second sends a request to the server, and the server replies with the current time, which then the application displays. When is it done rendering? Never. But Prerenderer needs to, at some point, decide that the page is done enough and ship it to the browser. […]

    […] Prerenderer will watch for events and once nothing happened for a period of time (300ms by default) it’ll consider the application done and if a certain amount of time went by (3s by default) even if the application is still active, it’ll stop and send it to the browser.

    That’s where the jump in 300ms is coming from and it’s constant. It’s not linear and definitely not exponential. It’s a constant number that can be tuned and tweaked. There are also some potential optimizations to reduce it or remove all together.

    The important thing is that all other values remained more or less the same and that the performance characteristics where quite stable. For me, this feels good enough to move on and start producing SPAs and with a bigger codebase we’ll be able to improve this library and make it better.

    Picture by Ian Farrel

  • Bidi vs Silk

    , , , ,

    In previous blog posts I mention that Bidi and Silk are essentially equivalent. I don’t believe this anymore. I now prefer Silk and I can show you why with a little example. First, let’s define some routes:

    (def silk-routes (domkm.silk/routes [[:home-page [[]]]
 [:about [["about"]]]]))

(def bidi-routes ["/" {"" :home-page
 "about" :about-page}])

    When it comes to defining routes, I find both a bit cryptic. Bidi feels a bit easier to read but I found it was harder to write in some scenarios.

    Continue reading →