Introducing wolkenkit 3.0

tl;dr: wolkenkit is an open source CQRS and event sourcing framework for JavaScript and Node.js. The new version, wolkenkit 3.0, includes a file storage service that was rewritten from scratch and that now supports authorization. The update also finally provides CLI commands to export and import your application data.

The past 6 months have been a highly exciting time for us. The wolkenkit CLI has been downloaded more than 3100 times. We have been to conferences and meetups in various cities, including Berlin, Bonn, Kassel, Munich, Nuremburg, Stuttgart, and a few other places. Additionally a number of talks were recorded, e.g. at the International JavaScript Conference (for a full list of recorded talks refer to the videos section of the documentation). Oh, and we had a guest post in our blog on commands and events by @marcusstenbeck. Thanks for that!

What also has made us more than happy is the increased number of great discussions with more and more people from our community. Additionally, we received a variety of valuable pull requests and feedback in general, e.g. by @nelreina, @nicolaisueper, @radumaerza, @reneviering and @zibur. As with the wolkenkit 2.0 release, we would also like to thank @scherermichael for his support in improving the wolkenkit release script.

Since we have been asked a number of times how to best get started if you want to contribute to wolkenkit, we now have a section in the documentation on contributing to wolkenkit, and we plan to extend this in the future. To improve transparency of what is planned and what we work an, we additionally made all issues public (and we now have a list of good first issues).

Introducing wolkenkit 3.0

Using the new file storage

As already mentioned, we rewrote depot, wolkenkit's file storage service, from scratch. When we had created the original version, we primarily focused on having a simple solution. Since then we have learned a lot about the needs of our users. As an example, we were asked several times how to protect stored files against unauthorized access.

Instead of trying to extend the simple first version, we decided to rebuild everything from scratch, this time with better knowledge of how people want things to work. The new version now has an authorization mechanism that integrates perfectly with the existing authentication and authorization features of wolkenkit itself. This lets you configure access rights for your stored files in a very fine-grained way.

Moreover, we have finally introduced a client SDK for depot, which makes accessing the file storage service as simple as possible. Of course, you can still use the low-level HTTP API directly, if you want to. With respect to this, we would especially like to thank @schmuto for providing very helpful feedback on storing large files!

To be ready for the future, we have also already taken into account that we are going to support other storage engines than the local file system, such as Minio and Amazon S3. Expect more to come here in a future release of wolkenkit.

Exporting and importing application data

No matter how hard you try, there are always things that go wrong. Unfortunately, wolkenkit 2.0 has contained a critical bug that we only discovered recently: Its CLI didn't pay respect to the --shared-key flag, which means that setting a shared key and persisting data did not work as expected. Actually, with wolkenkit 2.0, you are running at risk of losing your data when restarting an application.

That's what inspired us to introduce two new CLI commands, export and import. To export all the events contained in an application's event store use the wolkenkit export command. Complementary use the wolkenkit import command to restore the event store from a previously exported data backup.

The export is done as JSON files, which means that an export is not only useful as a backup, but can also be used to analyse events with third-party tools. Just think of what's possible by combining machine learning and events!

Finally, having an option to export and import data helps to make the transition from wolkenkit 2.0 to wolkenkit 3.0 a smooth experience. Please make sure to refer to the updating an application guide before migrating your application to wolkenkit 3.0 (otherwise, as said, you risk losing your application's data)!

Introducing a landing page

With wolkenkit 2.0, when starting a wolkenkit application, the CLI waits until the application becomes available by repeatedly sending HTTP requests to the /v1/ping route. Once an application was up and running, many people tried to access the API's / route, which unfortunately only replied with a 404 status code.

In the new version we now have a beautifully designed landing page, which turns accessing the API for the first time into a pleasant experience. Additionally, this landing page acts as the starting point for our long term vision of having a self-documenting API. Again, expect more to come here in future releases of wolkenkit.

Improving the write and the read model

With wolkenkit 3.0, there are also a few new features available to the write and the read model.

The write model became way faster when handling multiple that do not refer to the same aggregate. For consistency reasons it is necessary to handle commands that do refer the same aggregate sequentially. Until now this meant that all commands were handled sequentially, independent of the aggregate they referred to. Unfortunately, this made it possible that long-running commands could block the entire write model. Now, wolkenkit 3.0 runs up to 256 commands in parallel, while still keeping commands that refer to the same aggregate in order.

The read model has two new extension to the add function. The orUpdate extension allows you to add items if they did not yet exist – or to update them otherwise. In contrast, the orDiscard extension either adds an item – or does nothing, if it already exists. This lets you ensure in an easy way that an item is there either way.

Protecting credentials and other secrets

Finally, there is also a new feature available to the package.json file of your application. Since wolkenkit 2.0 it has been possible to add credentials and similar things as environment variables to your application, but their values were always visible as plain text. For obvious reasons, this is a bad idea for any sensitive data, such as credentials.

With wolkenkit 3.0 there is now an option to secure environment variables by introducing a separate file that is meant for storing secret values. We would like to say thank you to @go4cas for coming up with this idea!

It is needless to say that there are way more improvements and fixes, e.g. in the underlying eventstore. Thanks to @cessor and @domma for many highly interesting discussions on SQL!

To get a complete overview of what's new and updated, have a look at the changelog.

Updating to the new version

To update to wolkenkit 3.0, basically all you have to do is to install the new CLI using npm. The new CLI is fully backwards compatible and also supports the 1.0.0, 1.0.1, 1.1.0, 1.2.0 and 2.0.0 runtimes. What has worked before, will continue to work:

$ npm install -g wolkenkit

To update an application to the new runtime version, you need to make a few adjustments to your code due to the changes we have made to the API. We have described in detail what you need to do in the updating an application section in the wolkenkit documentation.

As said before, there has been a critical bug in wolkenkit 2.0, which may lead to data loss. If you have been using the --shared-key flag in the past, make sure to read the section on the danger of data loss!

Getting help

As always, if you need more details on how things work in wolkenkit, have a look at the wolkenkit documentation. If you have any questions or need help, you are welcome to join the wolkenkit Slack team. Also, there are a number of questions on wolkenkit on StackOverflow, so maybe your question had already been answered before.

Getting started

If you are new to wolkenkit, and version 3.0 is your first release, you may be interested in the brochure on CQRS, event sourcing, DDD and wolkenkit, which you can download for free as PDF (alternatively you may order professionally printed copies).

There is also a variety of articles available. We have also collected a list of blog posts and videos. And, there are a number of sample applications you may be interested in.

Conclusion

Finally, there is only one thing left to say: We are very looking forward to the future with wolkenkit, and as always – if you want so help us improve wolkenkit even further, feel free to contact us via email at hello@thenativeweb.io!

Thanks, and have a nice time with wolkenkit!

Yours, Sophie and the team of the native web

Twitter Facebook LinkedIn

Sophie van Sky

Cloud adventurer

We believe that openness is the foundation to build sustainable software. This very same principle we apply to technology. Hence we favor open standards over proprietary vendor-specific engineering.