Pablo Fernandez

Category: Technical

Posts related to programming or other highly technical stuff that can only be of interest to computer geeks.

  • Git-gc'ing all git repositories

    I was running out of storage space on my machine, so I started to search for things to remove using Grand Perspective. Some of the big files were inside Git repositories, or rather, inside the .git directory of those repositories. I decided it was time to run the Git garbage collector on them, all of them.

    I wrote this little script:

    #!/usr/bin/env bash

echo "Gitgcing $1"
cd "$1"

    and with this line:

    find . -name ".git" -type d -exec gitgc "{}" ";"

    run in my home directory, I got all my repos gc’ed.

  • Careful with that email

    When you are building systems like my Keep on Posting or my DNSk9 that send emails there’s always the danger that you’ll accidentally fire emails from your development machine to real users. You really don’t want to do that because it’s annoying and extremely unprofessional.

    It happened to me a couple of times. Thankfully, nothing serious. But I learned the lesson. That’s why in my user models now I have a safe_email method which I use instead of accessing email whenever I’m about to actually deliver a message.

    The method safe_email ensures that nobody will receive a message unless I’m in production and at the same time it’s good for testing. Obviously most of the time in development and testing mode I don’t deliver emails at all, but sometimes, I make an exception:

    def safe_email
  if Rails.env.production? || email.blank? # If the email is blank (or nil), let it be.
    email
  else
    "pupeno+#{email.gsub("@", "_AT_")}@pupeno.com"
  end
end

  • Data driven tests

    I’m not sure if anybody uses the terminology “data driven test” but if you explain what it is, experienced people will tel you that they are bad. Data driven tests are tests with the same code repeating over many different pieces of data.

    Let’s show an example. For my startup project Keep on Posting, I have a method that turns a blog url into a feed url. That method is critical for my application and there are many things that can go wrong, so I test it by querying a sample of real blogs. The test would be something like this (this is in Ruby):

    [sourcecode lang=”ruby”]
    class BlogToolsTest
    BLOGS_AND_FEES =>
    "http://blog.sandrafernandez.eu" => "http://blog.sandrafernandez.eu/feed/",
    "http://www.lejanooriente.com" => "http://www.lejanooriente.com/feed/",
    "http://pupeno.com" => "http://pupeno.com/feed/",
    "http://www.avc.com/a_vc" => "http://feeds.feedburner.com/avc",
    }

    def test_blog_to_feed_url
    BLOGS_AND_FEEDS.each_pair do |blog_url, feed_url|
    assert_true feed_url == BlogTools.blog_to_feed(blog_url)
    end
    end
    end
    [/sourcecode]

    Note: I’m using assert_true instead of assert_equal to make a point; these kind of tests tend to user assert_true.

    The problem with that is that eventually it’ll fail and it’ll say something like:

    [sourcecode]
    false is not true
    [/sourcecode]

    Oh! so useful. Let’s see at least where the error is happening… and obviously it’ll point to this line:

    [sourcecode lang=”ruby”]
    assert_true feed_url == BlogTools.blog_to_feed(blog_url)
    [/sourcecode]

    which is almost as useless as the failure message. That’s the problem with data drive tests. You might be tempted to do this an re-run the tests:

    [sourcecode lang=”ruby”]
    def test_blog_to_feed_url
    BLOGS_AND_FEEDS.each_pair do |blog_url, feed_url|
    puts blog_url
    puts feed_url
    assert_true feed_url == BlogTools.blog_to_feed(blog_url)
    end
    end
    [/sourcecode]

    but if your tests take hours to run, like the ones I often found while working at Google, then you are wasting time. Writing good error messages ahead of time help:

    [sourcecode lang=”ruby”]
    def test_blog_to_feed_url
    BLOGS_AND_FEEDS.each_pair do |blog_url, feed_url|
    assert_true feed_url == BlogTools.blog_to_feed(blog_url), "#{blog_url} should have returned the feed #{feed_url}"
    end
    end
    [/sourcecode]

    and if half your cases fail and the whole suit takes an hour to run and you have 1000 data sets you’ll spend hours staring at your monitor fixing one test every now and then, because as soon as one case fails, the execution of the tests is halted. If you are coding in a language like Java, that’s as far as you can take it.

    With Ruby you can push the boundaries and write it this way (thanks to executable class bodies):

    [sourcecode lang=”ruby”]
    class BlogToolsTest
    BLOGS_AND_FEES =>
    "http://blog.sandrafernandez.eu" => "http://blog.sandrafernandez.eu/feed/",
    "http://www.lejanooriente.com" => "http://www.lejanooriente.com/feed/",
    "http://pupeno.com" => "http://pupeno.com/feed/",
    "http://www.avc.com/a_vc" => "http://feeds.feedburner.com/avc",
    }

    BLOGS_AND_FEEDS.each_pair do |blog_url, feed_url|
    define_method "test_#{blog_url}_#{feed_url}" do
    assert_true feed_url == BlogTools.blog_to_feed(blog_url), "#{blog_url} should have returned the feed #{feed_url}"
    end
    end
    end
    [/sourcecode]

    That will generate one method per item of data, even if one fails, the rest will be executed as they are separate isolated tests. They will also be executed in a potential random order so you don’t have tests depending on tests and even if you don’t get a nice error message, you’ll know which piece of data is the problematic through the method name.

    Note: that actually doesn’t work because blog_url and feed_url have characters that are not valid method names, they should be replaced, but I wanted to keep the example concise.

    Since I’m using shoulda, my resulting code looks like this:

    [sourcecode lang=”ruby”]
    class BlogToolsTest
    BLOGS_AND_FEES =>
    "http://blog.sandrafernandez.eu" => "http://blog.sandrafernandez.eu/feed/",
    "http://www.lejanooriente.com" => "http://www.lejanooriente.com/feed/",
    "http://pupeno.com" => "http://pupeno.com/feed/",
    "http://www.avc.com/a_vc" => "http://feeds.feedburner.com/avc",
    }

    BLOGS_AND_FEEDS.each_pair do |blog_url, feed_url|
    should "turn blog #{blog_url} into feed #{feed_url}" do
    assert_equal feed_url, BlogTools.blog_to_feed(blog_url), "#{blog_url} did not resolve to the feed #{feed_url}"
    end
    end
    end
    [/sourcecode]

    and running them in RubyMine looks like this:

  • Rake tasks for production

    When I need to run something periodically on production, I always implement it as a rake tasks and install it as a cron job. Nevertheless there’s some setup to do in the task to have proper logging and error reporting.

    This is the template I use for creating those tasks:

    namespace :projectx do
  desc "Do something"
  task :something => :environment do
    if Rails.env.development?
      # Log to stdout.
      logger = Logger.new(STDOUT)
      logger.level = Logger::INFO # DEBUG to see queries
      ActiveRecord::Base.logger = logger
      ActionMailer::Base.logger = logger
      ActionController::Base.logger = logger
    else
      logger = ActiveRecord::Base.logger
    end

    begin
      logger.info "Doing something"
    rescue Exception => e
      HoptoadNotifier.notify(e)
      raise e
    end
  end
end

    While in development mode, it outputs to the console for convenience.

  • Another useful collection method? Enumerable#select_first

    For a personal project I’m working on, I need to find out the smallest time period with more than 5 records. I essentially wrote this code:

    period = [1.week, 1.month, 1.year].select_first do |period|
  Record.where("published_at >= ?", period.ago).count >= 5
end

    only to find out that the select_first method doesn’t exist. So I wrote it:

    module Enumerable
  def select_first(&predicate)
    self.each do |item|
      if yield(item)
        return item
      end
    end
    return nil
  end
end

    and then of course, I tested it:

    require "test_helper"

require "enumerable_extensions"

class EnumerableTest  2 }
  end

  should "select_first the first one" do
    assert_equal 1, [1, 2, 3, 4].select_first { |i| i >= 1 }
  end

  should "select_first the last one" do
    assert_equal 4, [1, 2, 3, 4].select_first { |i| i >= 4 }
  end

  should "select_first none" do
    assert_equal nil, [1, 2, 3, 4].select_first { |i| i >= 100 }
  end
end

  • A hash map method that returns a hash

    I’ve just released another gem, this one extends Hash to contain another method called hmap. This solves a problem I face ofter: how to run a map in a hash that returns another hash, for example:

    {:a => 1, :b => 2, :c => 3}

    being converted into

    {:a => 2, :b => 3, :c => 4}

    With hmap it’s easy:

    hash.hmap { |a,b| {a => b + 1} }

    It also works with arrays, but you must make sure the array you return always contains two and only two elements:

    hash.hmap { |a,b| [a, b + 1] }

    And that’s all, quite a simple piece of code, but now it’s re-usable and well tested.

  • A mistake for Nokia

    I think this partnership with Microsoft was a mistake for Nokia. It was great for Microsoft though.

    Every time you show an old Nokia phone you get the same comment: “Oh, those phones were built to last, I went through 3 iPhones and the Nokia still works…” or “That is the only phone that will not blend”.

    But Nokia’s obsolete software is killing it. They need to provide truly smartphones. They had three options:

    • build their own
    • use Android
    • use Windows

    I think they already tried to build their own and failed. That’s very hard. A giant like Microsoft tried to build their own and failed. It’s very hard to build a software platform.

    Apple did it by being the first ones and providing the coolest product ever. Google did it by doing it for free. They are both a success because they have thousands and thousands of applications on their platform by now.

    Microsoft haven’t done it. As powerful as Microsoft is, they still haven’t cracked the smartphone market, and it’s very likely they’ll never do. They are up against Apple and Google and both of them have years of advantage now (previous efforts by Microsoft are useless today).

    Now Microsoft has a chance to do it because of the deal with Nokia. Nokia is likely going to put Windows 7 Phone whatever on the hands of many people. Those people will get use to Windows, but not to Nokia and may switch to HTC or another provider in a blink of an eye. Microsoft wins, Nokia loses.

    I think Nokia should have done with Android. I know it’s hard to differentiate yourself with Android (what’s the different between a Samsung and a Sony/Ericson phone these days? They both run Angry Birds), but Nokia could have done it by making a tough phone. There’s a lot of people today not using smartphones because they won’t last in their pockets. Nokia could build a smartphone for them.

    Maybe Nokia decided against Android because of their past mistakes with Open Source projects and companies. At any rate, I think they are making a mistake right now.

  • Is this the new blue screen of death?

  • Faking it: enums in Ruby

    Ruby doesn’t have enums and in Rails I sometimes need it. I’ve come out with my own way of doing after some trial and error. First I want to be able to access these enums as constants in a class, like:

    Pizza::RAW
Pizza::COOKED
Pizza::BOXED
Pizza::DELIVERED
Pizza::EATEN

    That’s actually quite easy:

    class Pizza
  RAW = 1
  COOKED = 2
  BOXED = 3
  DELIVERED = 4
  EATEN = 5
end

    If I’m storing those values on the database, I’d like to have my database be more readable, so, I just store strings:

    class Pizza
  RAW = "raw"
  COOKED = "cooked"
  BOXED = "boxed"
  DELIVERED = "delivered"
  EATEN = "eaten"
end

    That’s not very efficient and there are better ways these days. But it’s simple and premature optimization is the root of all evil. Anyway, back to Ruby, I’d like to be able to get a list of all possible enum values, to be able to populate selections for example:

    class Pizza
  RAW = "raw"
  COOKED = "cooked"
  BOXED = "boxed"
  DELIVERED = "delivered"
  EATEN = "eaten"

  STATES = [RAW, COOKED, BOXED, DELIVERED, EATEN]
end

    Simple enough except that it doesn’t follow DRY style. In Ruby we can do better:

    class Pizza
  STATES = [
    RAW = "raw",
    COOKED = "cooked",
    BOXED = "boxed",
    DELIVERED = "delivered",
    EATEN = "eaten"
  ]
end

    That defines the constant and since the result of RAW = "raw" is "raw" we can also add it to an array at the same time.

    John, the manager of the Ruby Pizza Shop was concerned that some pizzas were devoured immediately and others took as long as 15 minutes. Decided to improve his business he started investigating the whole procedure and he noticed that as soon as the pizza left the kitchen, some cooks considered delivered and marked them as such. That was wrong so instead of doing a pizza-state training, he decided we should improve the UI. Let’s describe the state.

    class Pizza
  STATES = [
    RAW = "raw",
    COOKED = "cooked",
    BOXED = "boxed",
    DELIVERED = "delivered",
    EATEN = "eaten"
  ]

  STATE_EXPLANATIONS {
    RAW => "The pizza is not a pizza yet, just a bunch of ingredients.",
    COOKED => "OMG! That smells good!",
    BOXED => "It's ready to go.",
    DELIVERED => "The pizza has been snatched out of the hands of a delivery boy.",
    EATEN => "The pizza is no more."
  }
end

    Of course we can do better than that and merge it in one:

    class Pizza
  STATE {
    (RAW = "raw") => "The pizza is not a pizza yet, just a bunch of ingredients.",
    (COOKED = "cooked") => "OMG! That smells good!",
    (BOXED = "boxed") => "It's ready to go.",
    (DELIVERED = "delivery") => "The pizza has been snatched out of the hands of a delivery boy.",
    (EATEN = "eaten") => "The pizza is no more."
  }
end

    Stylistically I’m not a fan, but semantically, that’s dryer. Now, we can get the list of states like this:

    Pizza::STATE.keys

  • Metaprogramming Ruby

    There are thousands of books that will take you from illiterate to novice in any programming language. But finding those that will take you from novice or intermediate to expert is hard. I remember reading Effective Java some years ago and wishing I had something like that for Python. I’ve never found one.

    Metaprogramming Ruby is a great book full of very interesting knowledge, full of those things that separate a Ruby programmer and an export Ruby programmer. Before finishing the book I’ve already put to use some of the lessons and it saved me a lot of time. The book payed for itself before I’ve finished reading and I really recommend it to anyone who is serious about coding in Ruby.