Thursday, April 14, 2016

Interesting JSON Benchmarks Go.

These days lots of people are buiding microservices, and microservices usually involve HTTP API’s, which in turn usually exchange data as JSON.

Not long ago somebody pointed out that a lot of effort goes into generating and parsing that JSON. It would be unwise to simply ignore this part of your system’s design.

Since it’s very easy to benchmark things in Go, I decided to do a quick comparison of JSON encoding strategies.

Go JSON Encoding

The normal way to generate JSON in Go is to use the encoding/json package, and feed your struct into the MarshalJSON function. This function will take anything and try to convert it to JSON. If your struct, or anything in it, has its own MarshalJSON function then that is used, otherwise it’s examined using reflection.

Reflection is (supposed to be) expensive, so I wanted to see how much I might save by making my own JSON encoder for a struct. The main point being that I already know what the struct is made of, so I can save the encoder the trouble of examining it.

Benchmarked Variations

I started with several, er, structurally identical structs:

  1. A naïve one, with no MarshalJSON function of its own.
  2. A hinted one, with field names provided.
  3. A smart one, with its own proper MarshalJSON function.
  4. A fake one, which returns previously set data from its MarshalJSON.

The point of the fake one, of course, is to isolate the overhead of the actual JSON encoding.

All of these, when set up with a bit of standard fake data, generate the following JSON:

   "Id" : 123,
   "Stuff" : [
   "Desc" : "Something with \"quotes\" to untangle.",
   "Time" : "1970-01-01T01:16:40+01:00",
   "Insiders" : {
      "One" : {
         "Id" : 321,
         "Name" : "Eenie"
      "Two" : {
         "Id" : 421,
         "Name" : "Meenie"

Surprising Results

Here are the benchmark results for this little experiment, as run on a MacBook Pro (Mid 2014) with 2.8 GHz i5, 8 GB RAM, under Go 1.6.1.

BenchmarkNaïveJsonMarshal-4               300000          4299 ns/op
BenchmarkHintedJsonMarshal-4              300000          4293 ns/op
BenchmarkSmartJsonMarshal-4               200000          6490 ns/op
BenchmarkSmartJsonMarshalDirect-4         300000          4149 ns/op
BenchmarkFakeSmartJsonMarshal-4          1000000          2299 ns/op
BenchmarkFakeSmartJsonMarshalDirect-4   10000000           115 ns/op

In the “Direct” benchmarks, the struct’s own MarshalJSON function is called without going through encoding/json, i.e. without any sanity-checking.

I expected to see a lot of overhead from the reflection, i.e. the unknown struct being examined. Instead I found that using your own MarshalJSON function is actually slower because json.MarshalJSON (sensibly enough) validates the JSON output for you, lest it accidentally return invalid JSON itself.

Also, the hinting doesn’t make much of a difference, but it can make your JSON output prettier and more predictable: one usually uses it to have lowercase and/or underscore_separated key names in JSON objects, and to omit null objects in order to compact the JSON.

Using the numbers above we can very crudely estimate:

  • Custom encoding with validity checks is about 50% slower.
  • Custom encoding without validity checks is about 3.5% faster.
  • Best-case custom encoding with validity checks is about 50% faster.

In order to use the custom encoding without validity checks, you have to do all the encoding in a non-idiomatic way. This makes your codebase more fragile, because a new collaborator can’t just step in and do the obvious thing without undoing your optimizations.

It would be interesting to see how these numbers scaled with more complex structs, in particular deeper nested objects.

Based on these benchmarks, which I admit are oversimplified, I recommend avoiding custom MarshalJSON functions unless you absolutely need them for handling unusual data structures. If you want them for speed, make sure to benchmark your implementation before making a final decision.

Source Code

Labels: , , ,

Wednesday, April 06, 2016

I Am Possibly Legend

Recently I finally – finally! – got around to seeing the 2007 Will Smith SciFi vehicle I Am Legend. Apparently it was a remake, of sorts, of Omega Man, which I saw as a teenager and barely remember.

Spoiler Alert! Just in case you haven’t seen it.

Here’s the really big problem. The Zombies aren’t trying to kill Will Smith because he’s the savior of the human race, they’re trying to kill him because he’s a serial killer, responsible for hundreds of disappearances, tortures, and grissly murders. He’s more than a bad guy, he’s Zombietown’s own Josef Mengele.

He abducts the Zombie King’s girlfriend/daughter/wife/buddy/chess-partner (we aren’t told which), drawing the Zombie King out of his lair at risk of death by exposure. Our Hero then performs sick medical experiments on her, fully expecting her to die of them, in the name of “curing” her. She dies.

Then, the Zombie King – proving, by the way, they’re not zombies at all in the traditional film sense – comes up with an elaborate trap to capture Lt. Col. Mengele. After that fails, the Zombie King directs an army that almost kills Our Hero, but the Lt. Col. is saved by another NonZombie who shows up out of nowhere.

It should be said that Our Hero is at that point basically suicidal, because the Zombie King’s dogs zombified the Hero’s dog in their failed capture attempt, and the dog was Our Hero’s best and only friend, there being no other NonZombies around, and to top it off Hero had to kill Dog to prevent the zombification.

But! But all along our Hero makes audio notes, such as the audience is privy to, in which he says things about the Zombies that are demonstrably not true. Mostly, that they are actually zombies, and not merely Very Different Humans.

The only part of this I don’t understand is whether the filmmakers were trying to gloss over the murderous aspect, or whether they were through it trying to point out the futility of all life, of our folly before the gods. Because – spoiler alert – in the end the Good Dr. Lt. Col. Hero’s life’s work of abduction, torture and murder succeeds, and ends the plague of zombism.

Well, if you believe the narrator it does. Considering how reliable the narration is up to that point, I am much more inclined to believe there was a successful genocide launched against the Zombies. And for that matter I am specifically disinclined to believe they ate all the nice folks in the first place.


Saturday, March 12, 2016

Many Servers in One, with Go.

I’m still fiddling with a web server idea, currently trying my umpteenth version of it in Go. One of the things I want to do is serve a bunch of different things from one binary; they would usually live behind Nginx anyway, and I like the simplicity of serving a bunch of (somewhat) dynamic, small sites from a single program.

So, how does one do it? Why, one does it with Goroutines!

I had expected that to be the answer, but even then I was surprised at the simplicity of this solution. You don’t even need channels, since http.ListenAndServe is a blocking call if it’s not in a Goroutine.

Here’s the code, which should be self-explanatory.

Labels: ,

Friday, January 08, 2016

How to Test Logging in Go

Google’s Go language has a very rich and at times utterly infuriating standard library. One of the things it includes is Logging. Since there are six million ways to log (choose one!) we shouldn’t get too worked up about the ways in which the log package doesn’t log the right way for, well probably for anyone outside of Google. That’s not the point.

The point, and it’s a happy one today, is that a mere ounce of prevention will get you easily testable logging with Go’s standard log package. Caveat lector: I have no idea whether this works with Go prior to version 1.5.

Your Ounce of Prevention

Especially when hacking together quick utilities or packages, it’s always tempting to do this:

package logdemo

import "log"

func Log() {
    log.Println("woo hoo")

However, that will be very hard to test. Take the time to set up a logger, and expose it. Ideally you can do this with a struct:

package logdemo

import "log"

type LoggingThing struct {
    Logger *log.Logger
func New(name string) *LoggingThing {
    prefix := "thing-" + name + " "
    flags := log.Ldate | log.Ltime | log.Lmicroseconds
    return &LoggingThing{Logger: log.New(os.Stdout, prefix, flags)}
func (lt *LoggingThing) Log(msg string) {

Now you have something that can be messed with in your unit tests, and is also much easier to debug should the need arise.

But if you really can’t deal in structs, at least set up a package variable for the logger. This too can be manipulated.

package logdemo

import "log"

var logger_prefix = "global "
var logger_flags = log.Ldate | log.Ltime | log.Lmicroseconds
var Logger = log.New(os.Stdout, logger_prefix, logger_flags)

func Log(msg string) {

Your Pound Sterling of Cure

Now that we have an exposes Logger (two, really), we only need to know one thing: that a Logger can have its Output set after the fact, and that the Output needs only to be an io.Writer, which as you may recall simply requires that it implement this:

type Writer interface {
        Write(p []byte) (n int, err error)

In our test package, we can set up a simple struct to catch logs instead of writing them:

type LogCatcher struct {
    Logs []string
    Last string

func (lc *LogCatcher) Write(p []byte) (n int, err error) {
    s := string(p)
    lc.Logs = append(lc.Logs, s)
    lc.Last = s
    return len(p), nil

That will give us raw logs, but they may include prefixes and timestamps. Prefixes, if they are used at all, are quite useful; but timestamps are very hard to test for, since you then have to match log entries with regular expressions instead of simple strings.

Fortunately, that too can be overridden. Our overrides then look something like this:

thing := logdemo.New("one")

catcher := &LogCatcher{}

// If you want to test the prefix and output format you can of course
// do that separately.  For testing the written logs it's asking a
// lot, but we can override!

// Here we log, and catch it!
assert.Equal("here\n", catcher.Last, "caught first log")

Note that I am using the excellent assert package in order to make unit testing sane. (Again with the rich but infuriating standard library…)

If you’ve read this far and you aren’t totally lost then you can imagine how this continues. But you don’t have to imagine, because the code is there for you on GitHub under biztos/go-demos, specifically as
logdemo.go and logdemo_test.go.

Congratulations, now your logging will not prevent you from achieving the coveted (and utterly spurious) 100% code-coverage metric!

Labels: , , ,

Tuesday, November 03, 2015

Test Package Naming in Go

I’m still – still – playing with Google’s Go language, making the occasional one-off utility and also working on a Crazy Web Server Project. I hesitate to call it a Framework, since all Frameworks are ultimately doomed. It’s nothing too extreme, but it’s (hopefully) going to solve some of my web-server problems for small sites; and it’s (absolutely) helping me learn Go “for reals.”

I recently discovered something very simple, more or less by accident, that has big implications. It is this: tests for package foo should declare package foo_test.

To me this was counterintuitive, since one of the big hairy things you have to adjust to when learning go is the flat package directory. A package is made up of a bunch of .go files, and they all live in one flat directory. If you want to organize the files in a hierarchical set of directories, you must also have a corresponding set of packages (though they need not be hierarchical).

Incidentally, as I get better at Go I find myself hating such conventions less, because they are useful in forcing the programmer’s hand: be idiomatic damn you! And that certainly has its upsides.

Thus, logically, every .go file in a given directory declares the same package. If it doesn’t, you get a compile-time error!

Except, that is, for tests.

The Test Exception

For a directory with a package foo, you may have two packages: foo and foo_test. You may not have any others. So, why would you want to use foo_test?

There is a very good reason, which I’ll get to in a moment. If you already have a lot of unit-testing experience and know a little Go you may already have guessed.

Let’s first consider the way I had been doing it until recently. Apologies in advance for the lack of syntax highlighting.


package foo

func Bar() string { return "BAZ"}


package foo

import "testing"

func Test_Bar(t *testing.T) {

    exp := "BAZ"
    got := Bar()
    if got != exp {
        t.Errorf("Expected '%s', got '%s'", exp, got)

OK, great. This makes perfect sense so far. But now consider this, which in fact seems quite the obvious thing to do:


package foo

func Bar() string { return ook(1) }

func ook(i int) string {
    if i > 0 {
        return "BAZ"
    } else {
        return "BAT"


package foo

import "testing"

func Test_Bar(t *testing.T) {

    exp := "BAZ"
    got := Bar()
    if got != exp {
        t.Errorf("Expected '%s', got '%s'", exp, got)

func Test_ook(t *testing.T) {

    exp0 := "BAT"
    got0 := ook(0)
    if got0 != exp0 {
        t.Errorf("For 0, expected '%s', got '%s'", exp0, got0)

    exp1 := "BAZ"
    got1 := ook(1)
    if got1 != exp1 {
        t.Errorf("For 1, expected '%s', got '%s'", exp1, got1)


The example above shows a very common pattern, about whose virtue we could certainly argue: you can have much more thorough test coverage if you have robust unit tests for private functions. (Remember, in Go only Capitalized functions are exported.)

However, there’s a cost in clarity: you should be testing all things that could happen in the use of the package, i.e. you should be testing all variations of the public API. If there are conditions your public API can not trigger, and which thus can not be tested via the public API, then remove them from the code.

Go is strongly biased towards writing only the code that is actually used. Speculative code, while possible, is definitely not idiomatic.

As somebody who writes code for a lot of “what-if” scenarios in Perl, I can certainly apprecitate that discipline.

This is why you should use the foo_test package: Go will then throw a compile time error if you try to test a private function, and in your quest for 100% code coverage you will see the unreachable parts of your code.

So let’s get back to our example, changing the test first:


package foo_test

import (

func Test_Bar(t *testing.T) {

    exp := "BAZ"
    got := foo.Bar()
    if got != exp {
        t.Errorf("Expected '%s', got '%s'", exp, got)

If Bar is the only thing that ever calls ook then you’ll have to change ook or you’ll never get 100% coverage. In this contrived example, the compiler will most likely abstract away ook entirely, but bear with me.


package foo

func Bar() string { return ook() }

func ook() string {
    return "BAZ"

And there we have it, in silly little examples. Always use a _test package name for your Go tests. Testing will be harder. Deal with it: your tests will also be better.

Labels: , , , ,

Sunday, November 01, 2015

Never Pay Marhaba In Advance.

I recently flew to lovely, hazy Singapore, with a stopover in Dubai. It was a trip of many firsts: my first time in Asia, let alone Singapore; my first time flying Emirates; my first flight on an Airbus A380; and my first time in the famous Dubai Airport.

Since I had arranged for a four-hour stopover, I thought it would be nice to kill the time in a lounge. Since my lovely wife and I were traveling coach – not half bad on the A380 – the business lounges were out of the question. But fortunately there was a pay lounge avaialable: Marhaba.

After first getting a bite to eat elsewhere, we went to the Marhaba desk and asked about buying in. The attendants were quite friendly, and told us the lounge was rather full and thus we might not have a seat. I looked around, and while it was indeed full, there were some bistro-style chairs at a table. Good enough. We paid about 100 EUR for the two of us, and enjoyed the lounge.

I was planning on recommending it: it’s not as nice as a real business lounge, but it’s not too expensive and it beats just killing time in the airport. Plus, they mix a strong G & T.

However, the trip home went rather differently, and hence instead of a glowing recommendation I give you this advice:






Why not, you ask? Wouldn’t it be cheaper that way, you ask?

Well yes, it would be cheaper, except for the part about Marhaba keeping your money if your flight is delayed.

Our flight from Singapore to Dubai was significantly delayed after boarding. We sat on the plane for over two hours waiting for the engines to be replaced, or whatever technical defect they needed to fix (the Captain left us guessing). Then we were further delayed in the air; and just to make sure everyone was in the best possible mood, Emirates canceled the dinner service, at least in Coach.

When we arrived in Dubai, with clearly insufficient time to make our connection, we hustled through security and asked an attendant what we should do. He told us we should hurry. And so we jogged to the gate, and found the plane waiting.

About ten minutes later we were boarded, along with a few slower joggers, and on our way back to Europe.

Marhaba Will Not Refund Your Money.

I had booked the Marhaba lounge, pre-paid at a small discount, back in Singapore this time, as I figured it would be better that way. I had entered my flight number on their web site, so their system was fully aware of the delay.

Once back at a computer, I sent them an e-mail politely reminding them that my flight had been so delayed it would have been physically impossible to go to the lounge before making my connection, and asked them to refund my payment if they had not done so yet. (Honestly I half expected them to have issued a refund already, since they knew about the flight, but then I am a computer programmer.)

Marhaba got back to me promptly and told me that unfortunately, since I failed to cancel the booking eight hours in advance – i.e., since I failed to call them from the air – they would be keeping my money. Have a nice day.

I thought, well, that can’t really be your policy if you’re providing airport services, can it? This must be a pretty common occurrence: flight delayed, services can’t be used, at least issue a credit for the next time, right?

Bear in mind that lounge access is only one of the services Marhaba provides. You can easily shell out hundreds of Euros (thousands of Dirham) on Marhaba services, and then what happens when your flight is delayed, or for that matter canceled?

So I asked for clarification. And I got it:

We regret to inform you that no refund will be processed for a no show booking since any amendments or cancellation should be done at least 10hrs prior to the flight. Please be advised that all no show bookings has no refund.

Never Pay Marhaba In Advance.

Maybe that’s just how they roll in the Emirates. Maybe it’s usually somebody else’s money. Maybe everyone’s rich enough they don’t care about such things.

But if you care about your money, or about the principle of the matter, then it’s imperative that you never pay Marhaba in advance, for anything.

If I fly through Dubai again I might use their walk-in lounge service, since a strong drink is a strong drink, but my credit card shall never again darken the Marahaba website’s door. Neither should yours.

Labels: , , ,

Tuesday, October 27, 2015

Test Post from Classeur

Mostly nothing to see here folks, move along, move along…

This is just a test post from Classeur, which is a very slick in-browser Markdown editor with support for some popular blogging platforms.

I usually compose my blog posts here in Markdown first, then copy the generated HTML, then make a few corrections to get around Blogger’s buggy HTML-to-HTML converter, then paste it into the blogger UI.

Cumbersome, to say the least. Maybe this is a better way.

Also, maybe it’s an easy way to collaborate on content creation. Well, for open-source stuff it would be pretty easy to come up with a workflow, and the price is way right: $5/mo I believe.

But what if you’re, say, the editor of an online newspaper, and you want your writers to compose in Classeur?

The web site suggests you can publish to DropBox, which might be a solution, but that seems to not be implemented yet. Or you can share documents as read-write among Premium users, which would be a solution for getting it as far as the editor. The editor would then need to export it somewhere.

Yes, I realize Real Newspapers would have online workflows for all this, but first: I’m not a Real Newspaper, and second: I bet their workflows really really suck. Oh, and yes, I realize most online publications don’t employ editors, but humor me please.

So let’s assume this workflow is OK and will actually work:

  1. Reporter, who has a Classeur Premium account, writes a story.
  2. Reporter sends its link to the Editor, who also has a Premium account.
  3. Editor edits, if necessary sends it back, and so on until it’s ready.
  4. Editor exports the Markdown source and sticks it in a database, or source control, or whatever.

I like this so far. Not perfect, but the editor is really quite nice, and in my experience it’s not an easy thing to get non-techies to use plain text editors, much less parse Markdown in their minds.

The problem, though, is in the pictures. I can add a nice picture of a squash ballsquash ballsquash ball here:


The problem is that it’s on Imgur, an image hosting site very popular with the young and bored. And not a place I want my internet-breaking Paparazzi shots of the Donald.

But hey, if I’m a literary editor maybe I don’t need pictures. And if I’m just running some random click -baiting Bored Wombat site, maybe I don’t care.

Again: nothing to see here, test post mostly, &c., YMMV, E&OE, and the like. But I think Classeur is pretty interesting, and I will keep it in mind for future use.

Labels: , ,