Pablo Fernandez

  • Setting (database) credentials on a Spring Boot project, the right way

    , , , , ,

    Searching online for how to set up the credentials to access the database (or any other service) while in development leads to a lot of articles that propose something that works, but it’s wrong: putting your credentials in the application.properties file that you then commit to the repository.

    The source code repository should not have any credentials, ever:

    • You should be able to make your project open source without your security being compromised.
    • You should be able to add another developer to your team without them knowing any credentials to your own development machine.
    • You should be able to hire a company that does a security analysis of your application, give them access to your source code and they shouldn’t gain access to your database.
    • You should be able to use a continuous integration service offered by a third party without that third party learning your database credentials.

    If you want to see what happens when you commit your credentials to your repo, check out these news articles:

    That’s probably enough. I hope I convinced you.

    In an effort to find a solution for this, I asked in Stack Overflow and I got pointed in the right direction.

    Leave application.properties where it is, in your resources of code folder, commit it to the repository. Instead, create a new file in ${PROJECT_ROOT}/config/application.properties and also add it to your version control ignore file (.gitignore, .hgignore, etc). That file will contain the credentials and other sensitive data:

    # This should be used only for credentials and other local-only config.
spring.datasource.url = jdbc:postgresql://localhost/database
spring.datasource.username = username
spring.datasource.password = password

    Then, to help onboard new developers on your project (or yourself in a new computer), add a template for that file, next to it. Something like ${PROJECT_ROOT}/config/application.template.properties that will contain:

    # TODO: copy this to application.properties and set the credentials for your database.
# This should be used only for credentials and other local-only config.
spring.datasource.url = jdbc:postgresql://localhost/database
spring.datasource.username =
spring.datasource.password =

    And voila! No credentials on the repo  but enough information to set them up quickly.

    Disclaimer: I’m new to Spring Boot, I only started working with it a few days ago, so, I may be missing something big here. If I learn something new that invalidates this post, I’ll update it accordingly. One thing I’m not entirely sure about is how customary it would be to have ${PROJECT_ROOT}/config/application.properties on the ignore list. Please, leave a comment with any opinions or commentary.

  • My take on an AJAX re-frame effect handler

    There’s already an AJAX re-frame effect handler provided by Day8, the same guy who made re-frame and there’s nothing wrong with it. From the documentation, this is how you use it:

    [code lang=”clojure”]
    (re-frame/reg-event-fx ::http-post
    (fn [_world [_ val]]
    {:http-xhrio {:method :post
    :uri "https://httpbin.org/post"
    :params data
    :format (ajax/json-request-format)
    :response-format (ajax/json-response-format {:keywords? true})
    :on-success [::good-post-result]
    :on-failure [::bad-post-result]}}))
    [/code]

    That felt a little to verbose for my taste, so I made my own that you use like this:

    [code lang=”clojure”]

    (re-frame/reg-event-fx ::http-post
    (fn [_world [_ val]]
    {:ajax {:post "https://httpbin.org/post"
    :params data
    :on-success [::good-post-result]
    :on-failure [::bad-post-result]}}))
    [/code]

    If you are familiar with cljs-ajax, you’ll notice the pattern. Day8’s solution uses the raw cljs-ajax API, while I use the higher level one. Day8’s is more versatile, mine is more compact.

    This is my solution:

    [code lang=”clojure”]
    (ns ajax-fx-handler
    (:require [ajax.core :as ajax]
    [re-frame.core :as re-frame]
    [clojure.set :as set]))

    (re-frame/reg-fx :ajax
    (fn http-effect [request]
    (let [request-verb (set/intersection #{:get :post :put :patch :delete :head :options :trace :purge}
    (set (keys request)))]
    (if (not= 1 (count request-verb))
    (throw (js/Error (str "When specifying an AJAX request, one and only one verb should be specified. Found: " request-verb)))
    (let [request-verb (first request-verb)
    url (get request request-verb)
    request (if (contains? request :on-success)
    (assoc request :handler #(re-frame/dispatch (conj (:on-success request) %)))
    request)
    request (if (contains? request :on-failure)
    (assoc request :error-handler #(re-frame/dispatch (conj (:on-failure request) %)))
    request)
    request (-> request
    (dissoc request-verb)
    (dissoc :on-success)
    (dissoc :on-failure))
    ajax-fn (cond
    (= request-verb :get) ajax/GET
    (= request-verb :post) ajax/POST
    (= request-verb :put) ajax/PUT
    (= request-verb :patch) ajax/PATCH
    (= request-verb :delete) ajax/DELETE
    (= request-verb :head) ajax/HEAD
    (= request-verb :options) ajax/OPTIONS
    (= request-verb :trace) ajax/TRACE
    (= request-verb :purge) ajax/PURGE)]
    (ajax-fn url request))))))
    [/code]

    Feel free to use it, but, is it worth releasing it as a library? does anybody want that?

  • Pre-signed S3 URLs with Clojure and ClojureScript

    , , ,

    Jacob Emcken, the author of aws-simple-sign, reached out to me to share that his library can probably help with this, so if you need to achieve this goal, you might want to check it out.

    Disclaimer: I haven’t tried it or looked at its code.

    Dashman uses S3 for storing the encrypted screenshots. They can go directly from the renderer application to S3 and from there to the displayer application as there’s no need for the server application to ever see them. They are end-to-end encrypted anyway.

    My needs in this regard were a bit unique. I have a server component that can generate whatever tokens and signatures are needed and then send it to both applications. The applications are written in ClojureScript, so, essentially, JavaScript, but unlike most direct-to-S3 upload solutions, I don’t have a web site with a form, so, I don’t need to fiddle with CORS or forms.

    The two libraries

    I found two libraries that could have helped me. The big one is amazonica, a Clojure library to interact with AWS. It looks very complete but when I tried to use the com.amazonaws.services.s3/generate-presigned-url function, I got this error:

    The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

    There are two versions of signatures, version 2 and 4. Version 2 is deprecated, it might or might not work on some data centers but it definitely doesn’t work on all. I refused to believe it at first because I was using very recent version of all libraries, but yup, it was generating version 2 of the signature with signed URLs that look something like this:

    https://bucket-name.s3.amazonaws.com/key-name?AWSAccessKeyId=AKI...OA&Expires=1494190327&Signature=ct...3D

    Digging further I found this snippet of code:

    (System/setProperty SDKGlobalConfiguration/ENFORCE_S3_SIGV4_SYSTEM_PROPERTY "true")

    But for me, it made no difference in my case. For the record I reported the issue and it is supposed to work, but it doesn’t for me.

    Then there’s s3-beam, that looked promising, but it seems targeted towards forms, not just plain PUT requests from the clients. On top of that, I didn’t want to mess with async, I didn’t need it and it would be over-engineering for my case. Lastly, it uses it’s own custom way of creating the signatures instead of Amazon’s SDK and that code looks like it’s generating the old signatures (sha1 instead of sha256). I ended up not even trying it.

    Instead, I decided to do my own custom option, which was not hard and worked straight away. This post is about sharing that code.

    Custom solution using the AWS Java SDK

    Starting on the server/Clojure side of things, I added the dependency:

    [com.amazonaws/aws-java-sdk "1.11.126"]

    With only that I was getting this unexpected error when trying to use the AWS SDK:

    ERROR com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper;
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper;
ERROR Could not initialize class com.amazonaws.partitions.PartitionsLoader

    which I resolved by adding these other dependencies as well (mostly about getting the latest version):

    [com.fasterxml.jackson.core/jackson-core "2.8.8"]
[com.fasterxml.jackson.core/jackson-databind "2.8.8"]
[com.fasterxml.jackson.core/jackson-annotations "2.8.8"]

    I wrote a function to generate the S3 client:

    (defn aws-s3-client []
  (let [region (Regions/fromName (config/env :aws-s3-region))
        credentials (BasicAWSCredentials. (config/env :aws-access-key-id)
                                          (config/env :aws-secret-access-key))]
    (-> (AmazonS3ClientBuilder/standard)
        (.withRegion region)
        (.withCredentials (AWSStaticCredentialsProvider. credentials))
        .build)))

    config/env just fetches configuration values, powered by cprop and mount. This is just standard stuff from Luminus. With

    (defn- pre-signed-url [key & [verb]]
  (let [expiration (tc/to-date (-> 24 t/hours t/from-now))
        generate-presigned-url-request (doto (GeneratePresignedUrlRequest. (config/env :aws-s3-bucket-name) key)
                                         (.setMethod (if (= verb :put)
                                                       HttpMethod/PUT
                                                       HttpMethod/GET))
                                         (.setExpiration expiration))]
    (.toString (.generatePresignedUrl (aws-s3-client) generate-presigned-url-request))))

    The function generates either a URL for reading, :get, or for writing, :put, depending on the parameter verb. key is just the path part of the URL (remember, S3 is actually a key/value storage). All URLs will last for 24hs, something you can tweak as well.

    And that’s it, generating the URL is done, now we have to use it.

    Reading and writing from ClojureScript

    If you have a form, then, you might need to deal with CORS and other issues. I suggest you look at s3-beam at least for inspiration. For me, it was just a matter of generating the requests from the ClojureScript app, which uses cljs-ajax. Reading was straightforward enough:

    (ajax/GET get-url
          {:handler (fn [value]
                      (do-something value))})

    although the handler might have to deal with encoding issues. Writing is similar, but again, you may need to fiddle with encoding issues. Since what I was sending is a JSON payload (an encrypted screenshot), this worked for me:

    (ajax/PUT put-url
          {:format :json
           :params payload})

    If you are uploading binary data, I recommend you play with :format :raw and the :body attribute.

  • How to work with a private library in ClojureScript

    ,

    Dashman is composed of many components  including three desktop apps written in ClojureScript using Electron that share code through a private library (as in, it’s not open source).

    To have continuous integration and a set up that is easy to boostrap, that library should be deployed to a private repository. I achieved that using the plug in s3-wagon-private so that the project.clj of the common library looks like this:

    [code lang=”clojure”]
    :repositories [["private" {:url "s3p://s3-bucket-for-my-project/releases/"
    :username "not-telling-you"
    :passphrase "not-telling-you-either"
    :sign-releases false}]]

    :plugins [;…
    [s3-wagon-private "1.3.0"]]
    [/code]

    You should never have production or even staging credentials on your source code repository; but those credentials are neither. They allow to compile the project so, pretty much everybody that has access to the source code also needs access to that private repository. Having them in project.clj simplifies CI, on-boarding developers, etc.

    One you have that you can deploy the library by executing:

    [code lang=”bash”]
    lein deploy private
    [/code]

    The project of the apps using the common library look similar:

    [code lang=”clojure”]
    :repositories [["private" {:url "s3p://s3-bucket-for-my-project/releases/"
    :username "not-telling-you"
    :passphrase "not-telling-you-either"
    :sign-releases false}]]
    :dependencies [;…
    [tech.dashman/clientcommon "0.1.0-SNAPSHOT"]]

    :plugins [;…
    [s3-wagon-private "1.3.0"]]
    [/code]

    Yes… very similar.

    The next issue is compiling and re-loading code. You could modify the common library, install it, re-compile the apps, and re-run the apps. And you could do it while wearing a mullet and watching MacGyver (the original one). This is not the 80s! You want code reloading and all the goodies.

    If you are using Clojure, you should look into checkouts and maybe lein-checkout. In ClojureScript you can just add the path of the source code to your project even if it’s outside the project, with one gotcha. You might be used to a config like this:

    [code lang=”clojure”]
    :cljsbuild {:builds {:app {:source-paths ["src/app"]}}}
    [/code]

    which would imply this might work:

    [code lang=”clojure”]
    :cljsbuild {:builds {:app {:source-paths ["src/app" "../clientcommon/src"]}}}
    [/code]

    It might on non-Windows platforms, on Windows it doesn’t. On Windows you want this:

    [code lang=”clojure”]
    :cljsbuild {:builds {:app {:source-paths ["src/app" "..\\clientcommon\\src"]}}}
    [/code]

    Since you should probably never commit that anyway, it’s not a big deal.

  • ClojureScript debouncer

    ,

    I’m developing Dashman in ClojureScript (and Clojure) and as I was developing the cookie handler I found sites that constantly modify the cookies. As in, many times per second. Obviously I don’t want to encrypt the cookies and send them to the server that often, but even without taking that into account, just storing them to keep track of them in an atom, as in:

    (reset! cookies new-cookies)

    was slowing down the app considerably. I found myself needing to debounce so many calls as there’s no value in the middle calls, only the last one. Let’s backtrack a little bit here.

    The term to debounce comes from electronics:

    To remove the small ripple of current that forms when a mechanical switch is pushed in an electrical circuit and makes a series of short contacts.

    In programming it refers to a signal that gets emitted often and you want to process it but you don’t care about processing every intermediate value. To tie into electronics imagine you have a program that receives a message every time the temperature changes and the message contains all the temperature changes for the day. Clearly each message doesn’t have to be processed, only one, only the latest.

    Here’s where a software debouncer can help. It’s a function that gets a function and returns another function. When you call the resulting function 10 times, it will only call the original function once, with the last value it received.

    I found this debouncer written by David Nolan: https://gist.github.com/swannodette/5888989 and in the comments you can start to see that writing a debouncer might not be a trivial as you may think. One of the comments point to this other version: https://gist.github.com/scttnlsn/9744501. The comment implies this is a more correct version but I haven’t verified it myself. In that last page you can see someone found a bug and posted an improved version.

    For me, when a supposedly simple code snippet generates so many comments and versions, it’s better to find a library that implements it than to copy and paste and never maintain it again. Unfortunately, I couldn’t find a ClojureScript debouncer library but I got pointed to goog.function.debounce which you can use this way;

    (ns whatever
  (:require [goog.functions]))

(def debounced-println (goog.functions/debounce println))

  • The Highlander query with Rails’ ActiveRecord

    , ,

    For those cases in which there can be one and only one record on the database with certain fields and I don’t just want to get the first one and silently get the wrong one. I want to make sure there’s one and only one, so, I wrote this little extension to ActiveRecord that does exactly that:

    [code lang=”ruby”]
    module ActiveRecordExtension
    extend ActiveSupport::Concern

    class_methods do
    def one_and_only
    records = limit(2).all.to_a
    if records.count > 1
    raise "#{self} generated more than one record when expecting only one."
    else
    records.first
    end
    end

    def one_and_only!
    one_and_only.tap do |record|
    if record.nil?
    raise "#{self} didn’t generate any records."
    end
    end
    end
    end
    end

    ActiveRecord::Base.send(:include, ActiveRecordExtension)
    [/code]
    The first method, one_and_only, will raise an exception if there’s more than one item but it’ll return null if there aren’t any. one_and_only! will fail if there isn’t exactly one and only one record in the database.

    If you don’t know why I’m calling this The Highlander query, you should go and watch Christopher Lambert’s masterpiece.

  • Screensaver Ninja might be coming back

    ,
    ,

    Since discontinuing Screensaver Ninja, I have received many messages asking when it is coming back: over Twitter, Facebook, email, and even one person tracking me down on Reddit..

    For those of you who don’t know what Screensaver Ninja is, here is the old explainer video:

    https://www.youtube.com/watch?v=E2XStYc3TmQ

    It has been very painful to read these messages for a couple of reasons. Firstly, I strongly believe in the product. I want to have it; use it; and enable others to use it. I constantly see expensive and badly designed dashboards or wasted screens, which my product will address. Secondly, judging by the requests, other people want this just as much as I do. Not proceeding with Screensaver ninja could be a wasted business opportunity; although it is hard to tell if the demand is enough to support its development right now.

    I set up a landing page explaining what happened to Screensaver Ninja and a form for people to register for notifications of its potential comeback. This was a way to save everybody’s time and frustration; for those emailing requests for up to date information when I could only say, with sadness, “it’s over”. To my surprise, this form has been gathering five or so leads a week, which is rather a lot for an abandoned product.

    I have started playing with the idea that I might revive Ninja: This time I have designed a bigger system that covers many more use cases and allows me to support both Windows and Mac OS as well as other platforms just as easily.

    During this process I identified the technological bottlenecks; the aspects to product creation that can take months to negotiate and solve, such as hacking Apple’s cookie jar or packaging Chromium. In doing so, I have built a selection of prototypes testing my choices – and everything is working beautifully.

    So that’s it: I have decided to revive Screensaver Ninja. I have emailed all of you whom have shown interest to tell you the good news, and have received an overwhelmingly positive response from both individuals and corporations; some wanting to run hundreds of instances.

    I want to be completely transparent with my supporters; I am building Screensaver Ninja by myself in my spare time between long days and after hours work at two different consultant gigs. Whilst I am looking into the options of partnerships, developers, and marketers, I have decided not to wait for these additions to the team in order to make progress. I’m very excited about this phase both from the technical as well as the business points of view so Screensaver Ninja is moving forward and I will have frequent updates.

  • Going into the property business 

    , , , ,

    ​I’m a tech entrepreneur and that has not changed, but after six or seven years of trying to have at least moderate success, I’m starting to hedge my bets.

    On the side, I want to have some passive income and it looks to me like buy and hold properties, that is, buying them and renting them out, also known as buy to let, is the way to move forward. I identified some very profitable areas and they are in the least expected places. For example, in London you can expect a return on investment of around 3% while I’m getting something around 16%.

    If you want to listen to my journey, I’m documenting it as I go with Clayton Morris on his podcast about investing in property. He just published the second episode in which I talk about getting the money for my first two deals.  I just pulled the trigger on starting the business. I’m super excited and I can’t wait to share more good news in the next episode. 

  • Emacs is hurting Clojure 

    , , , ,

    Emacs is a very powerful text editor and its popularity amongst Clojurians is easily understood. Emacs has a long tradition in the Lisp communities as it’s written, in a big part, in a flavor of Lisp called Emacs Lisp.

    Because of its history, it handles Lisp code wonderfully, with automatic correct indentation, paraedit, integration with REPLs, etc. But Emacs is really hard to use.

    Yeah, most Clojurians know how to use it by now and they suffer from bias: “it’s not that hard” they say. Learning Emacs or Clojure is hard enough. Combining them is insane.

    Many Clojurians also say it’s worth it. And again, I think they are biased. Human brains are very good at forgetting pain. Other editors these days are also very powerful and although not as much as Emacs, their usage is intuitive so you can achieve a higher level of proficiency just by using it, without spending time and effort in becoming better at it.

    The way Emacs is hurting Clojure is by Clojurians maintaining this myth that you need to use Emacs for Clojure. This is not done by simple statements but by a general culture of jokes saying things such as “you are wrong if you don’t use emacs”.

    Me, personally, I don’t care what editor you use. If you want to learn Emacs, go for it. Intellij and Cursive is much easier to use and almost as powerful. When I compare myself to another clojurian, productivity is generally decided by knowledge of the language and libraries, not the editor. If you want to use another editor, so be it. It’s better if they understand Lisp code but it’s not a deal breaker for learning Clojure.

    I do care about the success and popularity of Clojure. Coupling the growth of the language to the use of an editor that is hard to use and non intuitive makes no sense. It’s hurting us. Even if you are an Emacs power user, when you talk to a Clojure newbie, please, don’t push it down their throats.

    Thank you.

  • Avoiding threads of emails when developing a Rails application

    , , ,

    Call to Buzz, like many applications I developed before, sends emails. Lot’s of emails. To avoid accidentally emailing a customer or random person I use mail_safe. It’s one of the first gems I install on a Rails project and you should too. mail_safe re-writes the to-header so you end up receiving all the emails that you sent.

    Once you are receiving all these emails, there’s another problem. They are likely to have exactly the same subject, so, mail clients are likely to group them in threads, which can be annoying. To avoid that, I added this to my ApplicationMailer class and voila! all emails have a unique subject:

    [code language=”ruby”]
    if Rails.env.development?
    after_action :uniq_subjects_in_development

    def uniq_subjects_in_development
    mail.subject += " #{SecureRandom.uuid}"
    end
    end
    [/code]