From Node.js to Go: There, and back again

tl;dr: A few years ago, we chose Go over Node.js for writing the wolkenkit CLI. A few months ago we rewrote everything from scratch, this time using Node.js again. We've learned that you can't dance at two weddings.

Command line interfaces (CLIs) have always played an important role in software development, as they are at the heart of any Unix-like system. Nevertheless, command line tools did not always have a good reputation.

In recent years, however, they have experienced an expansion, not least because of the increased demands with regard to automation and the constantly rising use of Linux on the server side. Therefore, this seems to us to be an excellent moment to tell about our experiences and findings during the development of the wolkenkit CLI.

Internally wolkenkit works a lot with Docker containers. The CLI serves as a kind of remote control that simplifies the handling of these containers. Having to write this remote control took us on an interesting and exciting journey from Node.js to Go and back again. We also learned a lot about how to integrate services via HTTP and CLIs respectively.

From Node.js…

When we started working on wolkenkit in 2012, we decided to develop it with JavaScript and Node.js. Therefore it was obvious to use the same technologies to develop the corresponding CLI.

Back then, we decided to use Docker's HTTP API because we thought it was the most reliable and stable way to interact with Docker. We hoped that, ideally, we could use an npm module that abstracts away these API calls for us, so that we could work with a nice Node.js API.

Unfortunately, it turned out that we were wrong: The HTTP API changed regularly, which is not surprising in retrospect. Docker was young then and still had to establish itself. Furthermore, the HTTP API was not designed for end users. Therefore, with every Docker update we had to adapt our code to keep it compatible with the new Docker version.

…to Go…

At that time we had to think about how we wanted to proceed. How could we let things mature while remaining agile and flexible?

The first important aspect for us was that we wanted to simplify the installation process of wolkenkit. At that time you had to install a lot to get things up and running, i.e. Docker, docker-machine, Node.js and the CLI.

As our installation instructions page grew, Go became more and more our focus. The ability to create statically linked binary files without any dependencies for multiple platforms at once seemed terrific. This seemed to be a much easier approach. Also, the proximity of Go to Docker seemed much better than that of Node.js to Docker: Since Docker itself is written in Go, we thought it would be much easier to address Docker with Go.

So we had to figure out how to interact with Docker from within Go. But unfortunately we hadn't learned our lesson yet: we assumed that the native Docker APIs for Go should allow a much better integration, but here too we should be enormously mistaken. Since we like learning new languages, it also seemed like a good way to learn more about Go and its approach to writing asynchronous code.

At the beginning we felt encouraged by the new stack: Go is a great language and the Docker API worked very well. But then the problems gradually began. To be clear here, the main problems we encountered were not related to Go as a programming language. Essentially, they were caused by our design decisions.

Docker also decided to split its API into multiple smaller modules and make them available individually. This of course entailed a lot of changes to our code, which we only managed to cope with to a limited extent. We had great difficulties in managing dependencies, especially as we were very spoiled by npm in this respect (which, by the way, does an amazing job at managing dependencies). In short, we were not prepared for the differences of the ecosystem of Go, in which various approaches to handle dependencies competed at the time.

So as we approached the first release of wolkenkit, work on our CLI slowed down. And again we were forced to make a decision: Should we rewrite the CLI from scratch once more?

…and back again

We weren't really happy, but we took the time to discuss different ideas and try things out. These ideas ultimately gave rise to a very simple idea: Isn't Docker's CLI the most stable API Docker provides? And couldn't we build on that by writing a CLI in Node.js that outsources the work to Docker's CLI using child processes? How would this affect our user experience?

So we decided to develop a spike and a quick proof of concept. We just tried to implement a single command to see how it felt. To our surprise, this has worked excellently in several respects.

Since we are very familiar with JavaScript and Node.js, it was easy to create the proof of concept. The performance of our first prototype was also promising: Remote controlling Docker via a child process was even faster than using the native APIs. This is probably due to our limited experience in Go, but it was still remarkable.

In addition, there were some other aspects that encouraged us to write the CLI once again in Node.js. On the one hand, Node.js had grown as a platform for writing CLIs. Thanks to modules such as command-line-args, setting up a modular and lightweight basic frame for a CLI has become very easy. On the other hand, we have also developed buntstift, a module to create consistent UIs for our CLIs. Last but not least, the installation of Docker on macOS and Windows has been dramatically improved.

Conclusion

Looking back on our journey, we have learned a lot. First of all, we have realized that Go is not JavaScript and that it is extremely difficult to master two technologies equally well. We also realized that integration via a CLI is an excellent option because CLIs are often very stable and well documented.

Additionally, end users are not interested in whether a tool has been developed with one technology or another. It is important to them that the product is usable, reliable and high-performance. So the most important thing for us is to achieve this goal and still make rapid and safe progress. For us this means that we build on our core competence with JavaScript and Node.js and trust that we will get all problems solved thanks to our experience.

However, this insight is also a result of our excursion to Go. We have learned a lot from this fast-growing ecosystem and will continue to explore it and others, because it never hurts to look beyond one's own nose. Nevertheless, it is also good to know where one's own roots are. The most important insight, however, is that developing software is ultimately not a question of one or the other technology or language, but of a system and its limits.

So, to cut a long story short: Have fun with the wolkenkit CLI!

Twitter Facebook LinkedIn

Golo Roden

Founder, CTO, and managing partner

Since we want to deliver elegant yet simple solutions of high quality for complex problems, we care about details with love: things are done when they are done, and we give them the time they need to mature.

Matthias Wagler

Lead core development

Discovering and developing things works better if you're not alone. Hence we pair for everything and always talk about what we think. We are deeply convinced that one and one makes more than just two.