Interesting JSON Benchmarks Go.

These days lots of people are buiding microservices, and microservices usually involve HTTP API’s, which in turn usually exchange data as JSON.

Not long ago somebody pointed out that a lot of effort goes into generating and parsing that JSON. It would be unwise to simply ignore this part of your system’s design.

Since it’s very easy to benchmark things in Go, I decided to do a quick comparison of JSON encoding strategies.

Go JSON Encoding

The normal way to generate JSON in Go is to use the encoding/json package, and feed your struct into the MarshalJSON function. This function will take anything and try to convert it to JSON. If your struct, or anything in it, has its own MarshalJSON function then that is used, otherwise it’s examined using reflection.

Reflection is (supposed to be) expensive, so I wanted to see how much I might save by making my own JSON encoder for a struct. The main point being that I already know what the struct is made of, so I can save the encoder the trouble of examining it.

Benchmarked Variations

I started with several, er, structurally identical structs:

1. A naïve one, with no MarshalJSON function of its own.
2. A hinted one, with field names provided.
3. A smart one, with its own proper MarshalJSON function.
4. A fake one, which returns previously set data from its MarshalJSON.

The point of the fake one, of course, is to isolate the overhead of the actual JSON encoding.

All of these, when set up with a bit of standard fake data, generate the following JSON:

{
   "Id" : 123,
   "Stuff" : [
      "fee",
      "fi",
      "fo",
      "fum"
   ],
   "Desc" : "Something with \"quotes\" to untangle.",
   "Time" : "1970-01-01T01:16:40+01:00",
   "Insiders" : {
      "One" : {
         "Id" : 321,
         "Name" : "Eenie"
      },
      "Two" : {
         "Id" : 421,
         "Name" : "Meenie"
      }
   }
}

Surprising Results

Here are the benchmark results for this little experiment, as run on a MacBook Pro (Mid 2014) with 2.8 GHz i5, 8 GB RAM, under Go 1.6.1.

BenchmarkNaïveJsonMarshal-4               300000          4299 ns/op
BenchmarkHintedJsonMarshal-4              300000          4293 ns/op
BenchmarkSmartJsonMarshal-4               200000          6490 ns/op
BenchmarkSmartJsonMarshalDirect-4         300000          4149 ns/op
BenchmarkFakeSmartJsonMarshal-4          1000000          2299 ns/op
BenchmarkFakeSmartJsonMarshalDirect-4   10000000           115 ns/op

In the “Direct” benchmarks, the struct’s own MarshalJSON function is called without going through encoding/json, i.e. without any sanity-checking.

I expected to see a lot of overhead from the reflection, i.e. the unknown struct being examined. Instead I found that using your own MarshalJSON function is actually slower because json.MarshalJSON (sensibly enough) validates the JSON output for you, lest it accidentally return invalid JSON itself.

Also, the hinting doesn’t make much of a difference, but it can make your JSON output prettier and more predictable: one usually uses it to have lowercase and/or underscore_separated key names in JSON objects, and to omit null objects in order to compact the JSON.

Using the numbers above we can very crudely estimate:

• Custom encoding with validity checks is about 50% slower.
• Custom encoding without validity checks is about 3.5% faster.
• Best-case custom encoding with validity checks is about 50% faster.

In order to use the custom encoding without validity checks, you have to do all the encoding in a non-idiomatic way. This makes your codebase more fragile, because a new collaborator can’t just step in and do the obvious thing without undoing your optimizations.

It would be interesting to see how these numbers scaled with more complex structs, in particular deeper nested objects.

Based on these benchmarks, which I admit are oversimplified, I recommend avoiding custom MarshalJSON functions unless you absolutely need them for handling unusual data structures. If you want them for speed, make sure to benchmark your implementation before making a final decision.

Source Code

Many Servers in One, with Go.

I’m still fiddling with a web server idea, currently trying my umpteenth version of it in Go. One of the things I want to do is serve a bunch of different things from one binary; they would usually live behind Nginx anyway, and I like the simplicity of serving a bunch of (somewhat) dynamic, small sites from a single program.

So, how does one do it? Why, one does it with Goroutines!

I had expected that to be the answer, but even then I was surprised at the simplicity of this solution. You don’t even need channels, since http.ListenAndServe is a blocking call if it’s not in a Goroutine.

Here’s the code, which should be self-explanatory.

How to Test Logging in Go

Google’s Go language has a very rich and at times utterly infuriating standard library. One of the things it includes is Logging. Since there are six million ways to log (choose one!) we shouldn’t get too worked up about the ways in which the log package doesn’t log the right way for, well probably for anyone outside of Google. That’s not the point.

The point, and it’s a happy one today, is that a mere ounce of prevention will get you easily testable logging with Go’s standard log package. Caveat lector: I have no idea whether this works with Go prior to version 1.5.

Your Ounce of Prevention

Especially when hacking together quick utilities or packages, it’s always tempting to do this:

package logdemo

import "log"

func Log() {
    log.Println("woo hoo")
}

However, that will be very hard to test. Take the time to set up a logger, and expose it. Ideally you can do this with a struct:

package logdemo

import "log"

type LoggingThing struct {
    Logger *log.Logger
}
func New(name string) *LoggingThing {
    prefix := "thing-" + name + " "
    flags := log.Ldate | log.Ltime | log.Lmicroseconds
    return &LoggingThing{Logger: log.New(os.Stdout, prefix, flags)}
}
func (lt *LoggingThing) Log(msg string) {
    lt.Logger.Print(msg)
}

Now you have something that can be messed with in your unit tests, and is also much easier to debug should the need arise.

But if you really can’t deal in structs, at least set up a package variable for the logger. This too can be manipulated.

package logdemo

import "log"

var logger_prefix = "global "
var logger_flags = log.Ldate | log.Ltime | log.Lmicroseconds
var Logger = log.New(os.Stdout, logger_prefix, logger_flags)

func Log(msg string) {
    Logger.Print(msg)
}

Your Pound Sterling of Cure

Now that we have an exposes Logger (two, really), we only need to know one thing: that a Logger can have its Output set after the fact, and that the Output needs only to be an io.Writer, which as you may recall simply requires that it implement this:

type Writer interface {
        Write(p []byte) (n int, err error)
}

In our test package, we can set up a simple struct to catch logs instead of writing them:

type LogCatcher struct {
    Logs []string
    Last string
}

func (lc *LogCatcher) Write(p []byte) (n int, err error) {
    s := string(p)
    lc.Logs = append(lc.Logs, s)
    lc.Last = s
    return len(p), nil
}

That will give us raw logs, but they may include prefixes and timestamps. Prefixes, if they are used at all, are quite useful; but timestamps are very hard to test for, since you then have to match log entries with regular expressions instead of simple strings.

Fortunately, that too can be overridden. Our overrides then look something like this:

thing := logdemo.New("one")

catcher := &LogCatcher{}
thing.Logger.SetOutput(catcher)

// If you want to test the prefix and output format you can of course
// do that separately.  For testing the written logs it's asking a
// lot, but we can override!
thing.Logger.SetFlags(0)
thing.Logger.SetPrefix("")

// Here we log, and catch it!
thing.Log("here")
assert.Equal("here\n", catcher.Last, "caught first log")

Note that I am using the excellent assert package in order to make unit testing sane. (Again with the rich but infuriating standard library…)

If you’ve read this far and you aren’t totally lost then you can imagine how this continues. But you don’t have to imagine, because the code is there for you on GitHub under biztos/go-demos, specifically as
logdemo.go and logdemo_test.go.

Congratulations, now your logging will not prevent you from achieving the coveted (and utterly spurious) 100% code-coverage metric!

Test Package Naming in Go

I’m still – still – playing with Google’s Go language, making the occasional one-off utility and also working on a Crazy Web Server Project. I hesitate to call it a Framework, since all Frameworks are ultimately doomed. It’s nothing too extreme, but it’s (hopefully) going to solve some of my web-server problems for small sites; and it’s (absolutely) helping me learn Go “for reals.”

I recently discovered something very simple, more or less by accident, that has big implications. It is this: tests for package foo should declare package foo_test.

To me this was counterintuitive, since one of the big hairy things you have to adjust to when learning go is the flat package directory. A package is made up of a bunch of .go files, and they all live in one flat directory. If you want to organize the files in a hierarchical set of directories, you must also have a corresponding set of packages (though they need not be hierarchical).

Incidentally, as I get better at Go I find myself hating such conventions less, because they are useful in forcing the programmer’s hand: be idiomatic damn you! And that certainly has its upsides.

Thus, logically, every .go file in a given directory declares the same package. If it doesn’t, you get a compile-time error!

Except, that is, for tests.

The Test Exception

For a directory with a package foo, you may have two packages: foo and foo_test. You may not have any others. So, why would you want to use foo_test?

There is a very good reason, which I’ll get to in a moment. If you already have a lot of unit-testing experience and know a little Go you may already have guessed.

Let’s first consider the way I had been doing it until recently. Apologies in advance for the lack of syntax highlighting.

foo.go

package foo

func Bar() string { return "BAZ"}

foo_test.go

package foo

import "testing"

func Test_Bar(t *testing.T) {

    exp := "BAZ"
    got := Bar()
    if got != exp {
        t.Errorf("Expected '%s', got '%s'", exp, got)
    }
}

OK, great. This makes perfect sense so far. But now consider this, which in fact seems quite the obvious thing to do:

foo.go

package foo

func Bar() string { return ook(1) }

func ook(i int) string {
    if i > 0 {
        return "BAZ"
    } else {
        return "BAT"
    }
}

foo_test.go

package foo

import "testing"

func Test_Bar(t *testing.T) {

    exp := "BAZ"
    got := Bar()
    if got != exp {
        t.Errorf("Expected '%s', got '%s'", exp, got)
    }
}

func Test_ook(t *testing.T) {

    exp0 := "BAT"
    got0 := ook(0)
    if got0 != exp0 {
        t.Errorf("For 0, expected '%s', got '%s'", exp0, got0)
    }

    exp1 := "BAZ"
    got1 := ook(1)
    if got1 != exp1 {
        t.Errorf("For 1, expected '%s', got '%s'", exp1, got1)
    }

}

The example above shows a very common pattern, about whose virtue we could certainly argue: you can have much more thorough test coverage if you have robust unit tests for private functions. (Remember, in Go only Capitalized functions are exported.)

However, there’s a cost in clarity: you should be testing all things that could happen in the use of the package, i.e. you should be testing all variations of the public API. If there are conditions your public API can not trigger, and which thus can not be tested via the public API, then remove them from the code.

Go is strongly biased towards writing only the code that is actually used. Speculative code, while possible, is definitely not idiomatic.

As somebody who writes code for a lot of “what-if” scenarios in Perl, I can certainly apprecitate that discipline.

This is why you should use the foo_test package: Go will then throw a compile time error if you try to test a private function, and in your quest for 100% code coverage you will see the unreachable parts of your code.

So let’s get back to our example, changing the test first:

foo_test.go

package foo_test

import (
    "./"
    "testing"
)

func Test_Bar(t *testing.T) {

    exp := "BAZ"
    got := foo.Bar()
    if got != exp {
        t.Errorf("Expected '%s', got '%s'", exp, got)
    }
}

If Bar is the only thing that ever calls ook then you’ll have to change ook or you’ll never get 100% coverage. In this contrived example, the compiler will most likely abstract away ook entirely, but bear with me.

foo.go

package foo

func Bar() string { return ook() }

func ook() string {
    return "BAZ"
}

And there we have it, in silly little examples. Always use a _test package name for your Go tests. Testing will be harder. Deal with it: your tests will also be better.

Test Post from Classeur

Mostly nothing to see here folks, move along, move along…

This is just a test post from Classeur, which is a very slick in-browser Markdown editor with support for some popular blogging platforms.

I usually compose my blog posts here in Markdown first, then copy the generated HTML, then make a few corrections to get around Blogger’s buggy HTML-to-HTML converter, then paste it into the blogger UI.

Cumbersome, to say the least. Maybe this is a better way.

Also, maybe it’s an easy way to collaborate on content creation. Well, for open-source stuff it would be pretty easy to come up with a workflow, and the price is way right: $5/mo I believe.

But what if you’re, say, the editor of an online newspaper, and you want your writers to compose in Classeur?

The web site suggests you can publish to DropBox, which might be a solution, but that seems to not be implemented yet. Or you can share documents as read-write among Premium users, which would be a solution for getting it as far as the editor. The editor would then need to export it somewhere.

Yes, I realize Real Newspapers would have online workflows for all this, but first: I’m not a Real Newspaper, and second: I bet their workflows really really suck. Oh, and yes, I realize most online publications don’t employ editors, but humor me please.

So let’s assume this workflow is OK and will actually work:

1. Reporter, who has a Classeur Premium account, writes a story.
2. Reporter sends its link to the Editor, who also has a Premium account.
3. Editor edits, if necessary sends it back, and so on until it’s ready.
4. Editor exports the Markdown source and sticks it in a database, or source control, or whatever.

I like this so far. Not perfect, but the editor is really quite nice, and in my experience it’s not an easy thing to get non-techies to use plain text editors, much less parse Markdown in their minds.

The problem, though, is in the pictures. I can add a nice picture of a squash ballsquash ballsquash ball here:

The problem is that it’s on Imgur, an image hosting site very popular with the young and bored. And not a place I want my internet-breaking Paparazzi shots of the Donald.

But hey, if I’m a literary editor maybe I don’t need pictures. And if I’m just running some random click -baiting Bored Wombat site, maybe I don’t care.

Again: nothing to see here, test post mostly, &c., YMMV, E&OE, and the like. But I think Classeur is pretty interesting, and I will keep it in mind for future use.

Go for Minor Utilities

I continue to play around with Google’s Go language, and I continue to have very mixed feelings about it.

It’s clearly not built for You and Me, i.e. for the programmers of the world; rather it’s built for Google and shared with us. This is fine as far as it goes, but as with so many other Google products the joy of the user is mostly “out of scope.”

Also, presumably because the whole Go ecosystem is so young, there are all kinds of quirky things being done in the many, many, many third-party packages scattered around the Githubbernets. I may write more later about the Goverse’s tendency towards Database Worst Practices and its shunning of the Null. (Ironic for a language in which every thirteenth line is a nil check.)

So far, I’m pretty sure I would not use Go for any big, complex, long-term project. But it’s good enough at some things that I’d definitely consider it for any straightforward, easy-to-run subcomponents of a big project. I wouldn’t choose Go for code shared with a bunch of other developers, but if I were dropped into such a project I’d probably do just fine. I might use Go if I planned on bringing in outside help, because as others have pointed out, one of the Google use-cases it seems optimized for is bringing inexperienced programmers into a project.

But there is one thing for which I absolutely love Go, and for which I could picture myself using it for a long time to come: Go is really great for writing simple utility programs.

Tonight I found myself needing some random-ish strings for a goofy little side-project I’m doing. After not finding quite what I wanted at the otherwise wonderful Random.org, I took an hour to throw together a little utility. It will probably only ever be utilitous to me, and I may or may not bother cleaning it up and testing it, but now at least it exists:

https://github.com/biztos/randstr

And that means I can, should I need to on some other computer, simply do this:

$ go get github.com/biztos/randstr
$ which randstr
/Users/Shared/gocode/bin/randstr
$ randstr --help

…assuming my $GOPATH is in order, and subject to dependencies I have not yet “vendored in,” and so on.

There are, I think, four things that make it especially attractive to write such programs in Go:

1. It compiles, quickly no less, into a single binary.
2. It’s reasonably cross-platform.
3. The standard library, while sometimes bizarre, is very rich.
4. Docopt!

The Joy of DocOpt

Command-line utilities, and Unix-like programs in general, usually take options and other arguments, and they usually have some rudimentary help text explaining these things.

DocOpt parses a structured, but very human-readable, help text in order to determine what the available options are. It’s not perfect but it more than covers the reasonable use-cases of most utilities I’ll ever need to write.

For any nerds who have not yet heard of DocOpt, here is the canonical example:

Naval Fate.

Usage:
  naval_fate ship new <name>...
  naval_fate ship <name> move <x> <y> [--speed=<kn>]
  naval_fate ship shoot <x> <y>
  naval_fate mine (set|remove) <x> <y> [--moored|--drifting]
  naval_fate -h | --help
  naval_fate --version

Options:
  -h --help     Show this screen.
  --version     Show version.
  --speed=<kn>  Speed in knots [default: 10].
  --moored      Moored (anchored) mine.
  --drifting    Drifting mine.

There are many implementations of DocOpt, but the one I have been using with Go is the excellent docopt-go by Keith Batten et al.

I do still love the completeness and flexibility of Perl’s Getopt::Long, and I’ve not found anything similar for Go; but now that I’ve gotten used to DocOpt I would think twice before committing to anything more complicated.

With DocOpt in your toolkit, Go is an awesome language for writing quick little utilities. Whatever else it may be, and whatever it may lack, this is reason enough to know some Go.

Go First Impressions

Over the last couple of weeks I’ve been playing with Go, an open-source programming language from Google.

My motivation was pretty simple: I felt like trying out a new server language, something modern and perhaps even trendy. Partly because it’s always a good idea to learn new things, and partly because I was growing frustrated with the limitations of Perl and Javascript/Node. Those are the two languages I use for most of my work, professional and otherwise. While I’d happily sing the praises of either one for many, many domains, every language has its baggage and I was lately feeling the weight of theirs.

Also, a highly respected member of the Perl community recently published a soul-searching article, The Mid-Career Crisis of the Perl Programmer. One thing it reminded me of is that I never considered myself a “Perl Programmer” any more than I consider myself an “Oil Painter.” I earn my keep as a software engineer. I have certain preferences and biases as would any serious professional, but I’d much sooner go to war over text editors than programming languages.

When I look into the crystal ball at my coding future, I see some Perl and some SQL (since 1994...) and that sure looks like Swift out there past the liquor store, but beyond that it’s a little fuzzy as we get closer to the beach. Maybe one of the fuzzies is a gopher, maybe not.

Without further ado, my first impressions of Go follow below.

Pro Go

Productivity Now!

I’m very impressed with how quickly I was able to be at least minimally productive in Go.

When trying out a new language, like most people I want to see if I can use it for something in the real world. In this case I tried a simple web application, as I’ve been writing a bunch of those lately.

A web app should be easy to prototype in this day and age, but it should also be obvious how you can go deep and scale up to a complex and high-performance system. That said, it doesn’t exercise features of the language so much as the available libraries.

After a week of part-time hacking and quite a bit of reading around the web, I feel like I know how to build a robust web app using Go and some of the popular libraries; but also, and here is where Go stands out, I’m fairly confident I could do the same with just the standard library. We’ll soon see why that’s important, but it’s worth noting that I couldn’t do the equivalent in Perl or Javascript without a lot of brute force, and I’ve been writing in those languages since before some Silicon Valley billionaires got to high school.

In fact I can’t remember the last time I played with a new language and so quickly found myself able to write real software, software I understand, and not some Thy-CRUD-Be-Thus templated Sinatrail tiger-trap.

This is a big deal: you might need to hire people, and assuming you are not an idiot you will probably hire people who don’t yet have expertise in the particular software stack you’re using, because that’s not a proxy for intelligence or productivity. (Hiring that way is, as we say in the coding trenches, an anti-pattern, like hiring only Ivy Leaguers or only those who can solve math puzzles in an interview.) These smart people you’ve hired need to get “up to speed” quickly, and it looks to me like Go is a language that would give one a leg up in that inevitable struggle.

The main criticism of Go I’ve encountered so far on the Web is that it may be a practical and a useful language, but it is also ugly and lacks this or that feature. This sounds about right, but for me ugliness exists on a sliding scale: show me the language that gets things done and I’ll show you the language that’s ugly and lacking some feature. Productivity is its own kind of beauty, at least for us Engineers.

Static Types.

As a Perl guy I didn’t expect to find myself saying this, but here it is: I’m starting to prefer static typing. It’s harder to do some things that way, but when I think of all the boilerplate crap I have to put into my dynamically typed code to make it reasonably safe, and all the tests I have to write to make sure I didn’t screw it up, static types start to look pretty good. In my experiments with Swift the static types have been more helpful than not.

And it’s not like you can’t deal with arbitrary types when you need to. Go makes it easy to write things like func DoSomething(m interface{}) {...} – though of course it would be a bad habit to do that very often.

What I’ve grown to like about statically typed languages is what we might call the enforcement of intent: if I write a function to turn Gophers into Marmots then the compiler or interpreter, by default, will make sure that I only ever give it gophers and that I know marmots, specifically, will be returned. This is probably as valuable for its encouragement of design clarity as for its bug-reduction potential.

Readability.

I wouldn’t say Go is a particularly beautiful language, in an aesthetic sense. But it is fairly expressive and once you get used to a few quirks it’s quite easy to read. This is obviously a key part of productivity: every time you write public static void or my ($self,$params) = @_ a poet dies.

You do have bad things, things that are ugly and non-obvious and will be the object of ridicule when Go is a bit older: map[string]reflect.Type is, for example, a perfectly reasonable return value in Go. But most of it is fairly obvious and easily grokable if you’ve done a bit of programming already. (And if you have not, maybe you should see the turtle before you talk to the gopher.)

Compiling and Statically Linking.

This is another one of those things that might sound odd coming from a Perl guy, but when I think about deploying Go programs on servers – possibly on very many servers – I really like the idea that my program, including all its dependencies, will be compiled into a single binary file.

That’s probably not the only file I’d have to deploy, but it removes a lot of hassle from my deployment story, whether that’s just me deploying to a Linode or a team of IT operations experts deploying to a server farm on SeaLand.

Sure, this introduces the problem that you need to compile on the same platform you intend to deploy on; but isn’t that what virtual machines are for? And if you’re not already testing on your destination platform then you’re not really testing, so to my mind this is the smallest of inconveniences. Even for a one-off little project you ought to ssh into your host for a final compile-and-test anyway.

Big Fat Standard Library.

One of the great strengths of Perl is CPAN, a vast repository of free software that will save you immeasurable effort in writing your programs. Go has nothing quite like that (see below); but it does have a very deep standard library. This means you can:

1. Build a lot without external requirements.
2. Have a common frame of reference in talking about Go programming.
3. Fall back to a reasonable midpoint if you find an external package buggy.

The frame of reference might be the most important part of this. If I’m building a web server I can use the net/http package and chances are very good I will be able to get help, should I need it, on sites like Stack Overflow. Because just about everyone who “knows Go” will know how net/http works.

Building software with fewer external requirements can be a great advantage in itself, simply because every dependency introduces a risk of bugs. If I find something genuinely wrong with net/http I am very confident it will get fixed, and that the fix will then be part of the standard software. If I find something wrong with github.com/biztos/gogogo/http then maybe the author will fix it, or maybe not; or maybe I’ll fork it, or maybe somebody else already forked it, or maybe I’ll just throw some sand into the wind and see where it lands, which from a stability point of view isn’t much less useful.

It’s also nice to know you can fall back if you need to, and not have to fall back all the way to sockets. How far this is realistically possible depends on the domain in which you’re operating, but one could at least bring up a decent web API without much external code beyond the database drivers. This might be irrelevant if you’re building the Twitter-Enabled Platform for Instagram Drivesharing (TEPID.IO, YC16). But if you’re stuck in that old-school world of solving actual problems with software it should help you sleep at night.

Scalable and Fast (they say).

I have not yet done anything at all with the features of Go that should, and purportedly do, make it scalable and fast. Unless you count compiling my code as a minimalist demonstration of speed:

$ go build hello.go
$ time ./hello
Hello world.

real    0m0.005s
use     0m0.001s
sys     0m0.003s
$ time echo Hello world.
Hello world.

real    0m0.000s
user    0m0.000s
sys     0m0.000s 

Hmm... But seriously, there’s a lot of chatter online about Go programs being pretty fast, and until I’m in a position to judge (not likely soon) I will take the Internet’s word for it. In any case it ought to be fast-ish just by virtue of its being compiled, right? Right?

The thing that looks most promising for higher-performance programming in Go is its concurrency model. I have not used this at all yet, but I like the look of it, and the fact that its keyword is go makes me think they gave it some serious thought. In fact it looks so promising that I find myself trying to think up a programming task for which I would need it. Some kind of batch processing I suppose...

Cloud Friendly, and Trendy!

What is cloud friendliness anyway? A buzzword, a buzzphrase; maybe it’s meaningless. But when you are thinking about writing things that might need “cloud scale” it’s worth asking how friendly the “cloud” is to your language. Amazon likes Go. Rackspace likes Go. Heroku tolerates Go, in a friendly sort of way. Google App Engine supports Go, in a tentative sort of way.

The concurrency model should make it fairly straightforward to write programs that scale with the size of a compute host, and that ought to help in cloud-style applications. But in the end the trendiness may be the key here: trendiness gets you support (Heroku is Trendy!) and trendiness gets you free software mashups with other trendy things. (The Cloud is Trendy!)

The other thing trendiness gets you is enthusiasm. This might sound shallow, but at the end of the day you will not be the only one writing your software. Having people excited about working with the latest, trendiest thing will improve their work. And that will make you money.

Standards You Can Live With.

With Perl, there’s always more than one way to do anything, but there are some established best practices for common scenarios. With Javascript, there’s usually more than one way to do anything, and there are dozens of competing practices all claiming to be the best. With Go, as far as I can tell, there is a consensus about the “right” way to do most things.

In coding patterns, the Go folks call this idiomatic programming, by which they mean that which follows the language team’s preferences. This is probably a good thing, but it’s hard to see how to enforce it other than to have an appointed Arbiter of the Idiom. That can work in open-source projects, where the Go Community will happily play that role. Internal projects are another story. As far as I know there is no equivalent of PerlCritic, but that hardly puts Go at a disadvantage to any language other than Perl.

Other standards are more actionable. For instance, unit testing is to be done with the built-in testing software. Code formatting is to be done with the gofmt command – a pale, sickly shadow of Perltidy but one that for better or worse brooks no argument.

Project layout also follows a standard, but it’s a very problematic one as I’ll discuss below.

In the end there is a “Go Way” and you can save a lot of time by just submitting to it instead of arguing about its merits. When this works, it’s probably worth having it not be perfect. I’m right about indentation and they are wrong, but if I have to prove I’m right to one more f---ing nerd then I might as well save the effort and live with their wrong indentation. When it doesn’t work, however, it leads to nonstandard standards, which is a slippery slope indeed.

Plan Nine!

Once upon a time, we computer types thought the world needed new and better operating systems. It turned out the world didn’t exactly need them: the Apple computer I’m writing this on, and the server rendering it for you, and the iPhone on which I’m listening to music right now, are all running their fancy software on top of UNIX, which is venerable and beautiful and stable and definitely not modern. Windows limps along apace.

Before the rise of Linux and Steve Jobs’ return to Apple, people really did think there would be something new running the world’s computers by 2015. One of the most exiting new things was Plan 9, named after a classic low-budget sci-fi movie.

It was brilliant and advanced and promising and it looked a lot like the future. But Linux rose, Steve Jobs used NextStep to save Apple, Microsoft stayed in business, and new operating systems like Plan 9 didn’t come to much.

But it was a great idea. And some of the same people who made Plan 9 are behind Go.

No Go

Living with Half-Baked Standards.

I’m sure there are others, but so far I’ve repeatedly been stymied by two poorly thought-out Go standards: project layout and packaging. The standard is in both cases so oddly incomplete that I wonder about the bubble in which it must have been created. Nice, comfortable bubble, no mess here, ah the purity of the bubble...

Project Layout

Go projects are supposed to live in workspaces, and a workspace has a few standard directories: src, pkg, and bin, with pkg holding compiled packages so you don’t have to recompile every last thing every time. You keep your workspaces in your $GOPATH environment variable, which supports multiple workspaces. Multiple workspaces can be useful in development if you want all your third-party packages in one place.

Here for example is a workspace in which I’ve built and installed the foo package, which has a foo/bar package within it and uses one external package, github.com/kr/text:

bin/foo
pkg/darwin_amd64/
                foo/boo.a
                github.com/kr/text.a
src/
    foo/
        main.go
        boo/booing.go
    github.com/kr/text/
                      .git/[shit-ton of git stuff]
                      License
                      Readme
                      doc.go
                      indent.go
                      ...et al. 

You will have noticed that the License and Readme files are thrown in with the source code, which is a bit messy but not unreasonable. And the shit-ton of git stuff is there because I fetched the text package with the standard go get command, and that clones the git repo.

So far so good, but what happens when I have a Real Life Project, in which I have a bunch of things that are essential but are not source code? Templates, configuration files, resources, test data. Where should it live?

Obviously I should have something like this, more or less, for a simple web app:

src/foo/main.go
src/bar/some/other/package.go
templates/index.tmpl
static/foo.css
config/foo-dev.conf 

As far as I can tell, that’s not what Go wants. Go wants me to do this, which is not wise:

src/foo/main.go
src/bar/some/other/package.go
src/foo/templates/index.tmpl
src/foo/static/foo.css
src/foo/config/foo-dev.conf 

I have yet to encounter any defense of this craziness, nor any official way to sanely lay out real-world projects in which the source code is but one part among many. I’m an optimist so I’ll keep looking, but so far it looks like it’s up to me to hack my way around Go’s deficient workspace concept.

If I’m lucky somebody smarter than me will come up with a great solution and it will become the de-facto standard. This often happens when somebody writes a popular Web framework and ships it with a sample application and/or an application generator, the default layout of which is carried forward by thousands of people using the framework.

But if we’re going to look for de-facto standards to emerge from the community, and in the meantime roll our own, then what is the point of having an Official Standard?

Packaging

Go’s default package manager is simply the go get command, which fetches a package from a public repository, does any necessary unpacking and installing, and drops the results under the first directory in your $GOPATH. It’s very clever about cloning Git repositories, and thus gives you a lot of flexibility when dealing with GitHub-hosted open-source packages, which seem to be the majority.

This is great up to a point. That point would be when you want to ship software that’s supposed to work, the not-working of which would be your problem, and which has external dependencies. Then suddenly you realize – oh shit! – you forgot to specify the exact version of that cool parser package from GitHub. And it’s at 0.31 so even assuming malice and human error do not exist in the world, your code still might break when cool/parser hits 0.33 and the API changes.

I keep hoping I will run across some justification for this, since it’s such an obviously bad design yet Go was written by obviously smart people. I have read that “stable code on HEAD” is the answer, but whoever says that doesn’t understand the question. It’s not about hygiene, it’s about predictability. The official justification isn’t much help either.

My best guess is that Go was written in a context, in-house software development at Google, in which package versioning was not an issue, and as a result the authors simply didn’t think of it.

Standard Anarchy.

One of the wonderful things about the open-source world, especially in this age of cheap internet infrastructure, is the way problems are often solved for you before you are even aware of them.

In the case of Go, we have GoPkg.cc, which very elegantly solves the package versioning problem for packages hosted on GitHub (the majority, I believe).

We also have GoDoc for package discovery and documentation. Between these two things Go almost has a CPAN.

I get the sense that Go revels in its anarchy at the moment, while also trying to insist that there are standard ways of doing most things. Presumably something like GoPkg will become a standard, de-facto or otherwise, in time.

The problem I see with this mix of anarchy and imperative is that not everyone is going to agree on the original standards. This problem grows with the user base, particularly as older, more experienced (and more opinionated) programmers get involved. The standard documentation, for example, is ugly and hard to use. Let’s say I make a better documentation package: should I refrain? Should the best solution win, or the factory standard?

Much has been made of the time saved by having gofmt format source code in the One True Standard Way, thus obviating the need to argue about indentation. But if there’s no standard manager for versioned external packages, and no standard Posix-compliant option parser, then why shouldn’t I make a competing code formatter? Let’s say I do, and it catches on. This could be a very good thing, but reduces the point of gofmt to simply saying “automated code formatters are good.” We’ve been saying that in the Perl world for a long long time.

Objects, Not Objects, Deja Vu!

I almost put this in the Pro Go section, because after half a million years writing object-oriented code I welcome the challenge of not thinking in “OO” terms, and I very much look forward to learning more about good objectless design. And anyway Go is kinda-sorta object-y to begin with.

In the real world, however, somebody will devise a clever hack that messily glues object inheritance onto Go’s structs and interfaces, which is where its non-inheriting yes-and-no object-orientation already lives.

In the real world, this object hack will be derided as nonsense by Go purists, will be more or less disowned by the Go community, will be exposed as borderline fraudulent in six dozen blog posts and half that many conference presentations, and will nonetheless be used, largely out of sight, in thousands of Go projects.

I think it was a big mistake of the Go authors to not build in proper object orientation. I don’t see why they couldn’t have. I get that they don’t want people to write Go that way, but sadly I believe a lot of people eventually will; and there won’t be any official way to do it, so somebody’s unofficial solution will end up as the base of some large and indispensible package that sooner or later everybody, or at least you, has to use.

Google: The Glass is Half.

To paraphrase jricher42’s Reddit comment: Google is an advertising company that realizes it must produce excellent software in order to continue dominating the advertising market.

It would be a little ridiculous to expect such a company to over-commit to any particular technology. If you’re a Go lover, and particularly if you’re one who doesn’t love Java and Python, then you probably wish Google would “Go” all-in and make Go their primary language. Then the outside world would love Go even more, because after all it’s good enough to run Google; and Google would get a bunch of free training for potential engineering hires.

But that’s not how this works, and for Go’s sake that’s mostly a good thing. First, given that Google’s support of Go is a little tepid even now, I wouldn’t assume it has any special prominence in Mountain View. As recently as last year I had a Google recruiter pitch me without once mentioning Go: he said I could code in the language of my choice as long as it was Java or Python. Second, Google for obvious reasons insists on the flexibility to discard its experiments, no matter how much “the community” might care about them (viz. Reader), and for that reason alone it’s better if Go isn’t a “Google Thing” per se. Third, Google is every bit as closed-source and poker-faced as Apple when it comes to any software that actually makes it money. If Google comes to depend on Go, then the internal Google fork of Go will start growing proprietary features if it hasn’t already. Good for Google, bad for Go.

This adds up to a risk, and one that we must hope Go outgrows. Google is capricious, and exerts an influence on the language. It might overcommit, it might meddle, it might abandon all support of Go; and in each of these cases the language might suffer, to an unknown degree.

Youth Itself.

Go is a very young language. It’s been publicly available since November 2009, which makes it five years old as of this writing.

Five years is time enough to mature some but not really time enough to prove any staying power. These have been faddish years in computer languages, and while Go’s popularity is growing it’s hard to guess how far it’ll, er, go.

Consider Perl: so very not trendy, yet so well established that an enormous amount of very serious code is written in it every day, making a lot of people a lot of money (and the programmers some too). Most of it isn’t powering startups, and I suppose none of it is powering Google, but Perl remains a reliable industry workhorse. I would much sooner trust Perl’s DBI than anything gluing databases onto Go.

Will Go become that reliable? Will an investment in Go expertise pay dividends in ten years? In twenty? Will there be a canonical Oracle driver, Perforce and Atlassian integration, that sort of thing?

I like Go largely because it solves problems I actually have, and the “canonical Go program” is a web application, of which I expect to write many more in years to come. But Go’s youth is a risk and will be for at least a few years. It would be helpful to hear success stories about large-scale adoptions of Go at major companies other than Google. Are there any such stories?

Limited Sense of Humor.

Much like Google itself, I get the feeling that desipite its cute (?) gopher mascot the Goosphere takes itself a bit too seriously. The Perl folks are much wittier.

Case in point:

$ go ogle
go: unknown subcommand “ogle”
Run ’go help’ for usage. 

Further Reading

If you’re interested in using Go, I recommend you start by playing with it, or following the official tutorial, or reading a free online book. If you’re the learn-by-doing type that will probably be enough to get you excited or repulsed, depending on whether Go is a natural match for you.

I also strongly recommend reading a talk the Go authors published in 2012, Go at Google: Language Design in the Service of Software Engineering. That explains the why of Go as well as much of the how,
and I don’t think the FAQ really makes sense until you’ve read it.

Finally, I found it enlightening to read a few “Go vs X” writeups, such as Scala vs. Go on Quora.

Now go build something!

Merging the Apps, Merging the App Stores

Apple’s recent World Wide Developer Conference revealed many exciting new software technologies, about which I will surely write more at some point. One interesting thing is that there is much better integration and communication between iOS and OSX on the horizon.

This got me thinking: why are there still two Apple App Stores?

(Why are they so awful is another question, and out of scope for now.)

Right now you can make an app for iPhone; or for iPad; or for both iPhone and iPad; or for the Mac. You would usually use the same tools to make any of these.

Why not merge the stores, and also merge the apps?

Merging the stores is a fairly obvious idea. It would mean using the "normal" (Mac) App Store to buy and manage mobile apps on your Mac. That might be a nice opportunity to get mobile devices out of iTunes once and for all, but again: out of scope.

Merging the apps is perhaps less obvious, but more and more iOS apps are spawning Mac versions. Wouldn’t it be cool if the developer could sell all three versions as one package? From the developer’s point of view it would probably involve uploading separate apps and simply merging them administratively. OSX and iOS are not likely to be the same thing any time soon, so in terms of binaries we are still talking about two apps.

But in terms of user perception we are talking about one app with three incarnations. I think people would get used to this very fast, and come to think of a good app as being available on all their devices automatically.

Not every developer would take advantage of this at first. Consider the Omni Group, one of the best independent software companies in the Apple ecosystem. They make some great software and they are not shy about charging real money for their products. I have one of their apps, OmniFocus for iPhone, iPad and Mac, and I paid three times for the privelege. Now they have released a new version as a paid upgrade (still pending for iPad) and I will probably, sooner or later, pay them three times again, because I believe I get sufficiently premium value from these premium products to justify their premium prices.

But for smaller developers looking to maximize their users’ engagement, it could be a big win. Or for Silicon Valley startups trying to metabolize their large injections of venture capital and desperate for eyeballs lest their Social Meatball World be overtaken by Meatball Cloudface LLC. You need to get on as many devices as possible, and get people to use your software as much as possible, before the Samwer brothers come out with Wolkenfleischball Sozial Total and lock you out of the German market.

As a user and a customer, I would like that. I want all my apps in one place, at least administratively. I want one "Purchased" tab. I want things that interoperate between my devices. I want to open iTunes as infrequently as possible.

Apple wants 30% of everything. Should be an easy match, no?

As a developer, however dilettantish my engagement with XCode may be, I find it exciting to think about making apps that are integrated out of the "box," as it were. Imagine, for example, an online banking app. Some things you really want to have available when you’re out and about, and some things are much easier to do on a "real" computer. Or think about all the things you can do (soon) with Apple’s new Health app. This begs for a desktop/laptop companion app.

I think Apple will merge the stores at some point. I have no idea when, and it would be silly to speculate considering I’ve seen iTunes as a dead man walking for years now. But sooner or later I think we will have one app store, at least for Macs.

Will they allow the apps to be merged as well? I hope so. I see a lot of benefit to it. But I could also imagine this waiting until some day, probably at least five years from now, when the operating systems themselves merge and you could have a three-platform app (or four, with the TV) that really is a single app.

Merged or not, I think the future of a lot of app categories is tight interaction between the phone, tablet and desktop/laptop versions. Doing this across platforms is much more challenging, and I have a hard time imagining the integration ever being very good. That might give Microsoft a much-needed boost to its mobile business, because at the moment only Apple and Microsoft are capable of tight integration throughout their respective ecosystems.

If I’m going to be so rash as to predict something, then it's this: by this time next year there will be a bunch of indie apps offering a unified user experience on iPhone, iPad and Mac, and at least a couple will be popular enough that the conversation in the developer community will be largely about that integration.

CanoScan LiDE 200 on Mountain Lion: Failed to open a Session on the Device because it (the Device) was Locked!

Just a very quick tech note for the Community, since it took me way too long to solve this problem: assuming you have updated your drivers and so on, if your CanoScan LiDE 200 (or similar Canon scanner) still gets you an error in Image Capture on a Mac, the scanner might be locked.

The lock switch is on the bottom of the scanner.

I had by then already updated the drivers, but I probably didn’t need to. Unlock; unplug; re-plug; re-launch Image Capture; voila. It took a while to warm up, but then it had been in storage for two years (which is of course why it was locked).

That did the trick for me. Of course it would have been lovely to have the software say “Your Scanner is Locked, Dumbo!” But I guess it’s a more secure lock if you forget it’s there!

UITapGestureRecognizer and “unrecognized selector sent to instance” errors.

This is just a very quick note for search engine posterity, in case anybody has the same frustrating problem I had today.

I am implementing some gestures for an iOS art app (as in a work of media art, the "medium" of which is an iPhone/iPad application). I was following the "Event Handling Guide for iOS" from Apple.

After much Googling and some very informative stops at StackOverflow, I had got basically nowhere. Everything seemed right, my code conformed tightly to the tutorial and to everyone else’s, and everybody else who found a solution had obviously solved a very different core problem than I was facing. What could be wrong?!

It turns out the problem was as simple as:

-(void) handleTap
{
    NSLog(@"*** GOT A TAP ***");
}
    

Now, in real life you would almost never use a method as dumb and simple as that, but when setting up your classes for initial hacking and debugging you just might. According to the tutorial, it should work:

The action methods for handling gestures—and the selector for identifying them—are expected to conform to one of two signatures:

- (void)handleGesture
- (void)handleGesture:(UIGestureRecognizer *)sender

Silly programmer, tricks are for kids! The standard-issue selector looks for only the second signature, and if it doesn’t find it you get the error whenever you tap: unrecognized selector sent to instance! boo!

The solution, then, is to declare and implement the method with the argument. That works fine.

-(void) handleTap: (UIGestureRecognizer *)sender
{
    NSLog(@"*** GOT A TAP ***");
}
    

To be fair, Apple has an enormous amount of documentation available online, and it’s not so shocking that a tutorial is out of date. For us iOS newbies it’s frustrating, but I don’t think anybody else is doing a better job of this at the moment.

Perl ExtUtils::MakeMaker test overrides (for Makefile.PL).

Nerd post follows; non-nerds, move along.

This took me a while to figure out, and in the end I had to dig around in the source code to figure out just what was going on, so here’s a quick note for posterity since I’m sure somebody else will have exactly this problem at some point. Caveat lector: I’m probably making a bunch of mistakes here. And thanks to brian d foy for showing us how to use test_via_harness.

Basically what it boils down to is this: you are going to have either a bunch of test files in a t directory, or a single test.pl file in the main directory. Or possibly both.

The thing you need to override is different in each case. You might just want to do both in case you change your mind later about the test setup.

Let’s say, for instance, that my special little requirement is that I can get at some shared libraries in the /opt/special/lib directory. The examples below should be understood as code inside your Makefile.PL file. And yes, they are almost identical in this case.

For test.pl override test_via_script.

sub MY::test_via_script {

    my ( $self, $perl, $script ) = @_;

    # Here is some special sauce for the environment:
    my $env_setup = 'LD_LIBRARY_PATH=/opt/special/lib';

    # Here are some Makefile variables that are handled by `make`:
    my $harness_args =
      '"$(TEST_VERBOSE)", "$(INST_LIB)", "$(INST_ARCHLIB)"';

    # Here's the command that will create the target:
    my $command =
      sprintf "\t%s %s -MExtUtils::Command::MM -e 'test_harness(%s)' %s",
      $env_setup,
      $perl,
      $harness_args,
      $script;

    return $command;

}

For t/*.t override test_via_harness.

sub MY::test_via_harness {

    my ( $self, $perl, $tests ) = @_;

    # Here is some special sauce for the environment:
    my $env_setup = 'LD_LIBRARY_PATH=/opt/special/lib';

    # Here are some Makefile variables that are handled by `make`:
    my $harness_args =
      '"$(TEST_VERBOSE)", "$(INST_LIB)", "$(INST_ARCHLIB)"';

    # Here's the command that will create the target:
    my $command =
      sprintf "\t%s %s -MExtUtils::Command::MM -e 'test_harness(%s)' %s",
      $env_setup,
      $perl,
      $harness_args,
      $tests;

    return $command;

}

Etc. & So On

I think it’s a much better practice to keep your test files in t/ anyway, and to always run with the test harness. In fact I usually skip the explicit test harness altogether and run everything with the appropriate prove command (the one that belongs to the Perl I’m using).

However, if you’re dealing with legacy code or not interested in doing real Perl stuff or otherwise just need to “get on with it” then you might well need a test.pl – and remember, test_via_script for that guy!

Here are the obvious links you should follow if you want to know more about this:

• ExtUtils::MakeMaker
• ExtUtils::MakeMaker::FAQ
• Chapter 10 of Mastering Perl by brian d foy.

And finally, for anyone nerdier than I who is reading this: I know about package MY and I agree it’s probably the better solution, but I think it’s (even more) confusing for novices; and I assume at least some people trying to do this aren’t even Perl programmers.