Testing Bun's compatibility with Node.js and speed with a complex application

; Date: Sat Jul 09 2022

Tags: Node.JS

Recently a new JavaScript server-side runtime, Bun, was announced, promising a massive speed improvement over Node.js. Instead of testing with a small application, let's try with a complex app to see real-world performance.

NOTE IMPORTANT UPDATE, READ BEFORE THE REST OF THIS ARTICLE

It appears that my test below is incorrect. In the version of Bun used in the test, I ran my application using cli.js which has a “shebang” at the front of the file, a.k.a. #!/usr/bin/env node. According to a comment below by Jarred Sumner, Bun respected the shebang, and therefore my code was executed by Node.js rather than by Bun. That's because the shebang says to use Node to execute the script.

I have prepared a followup article that more carefully runs some performance tests while repeating some of the observations discussed below. See: Deeper testing of Bun's performance and compatibility against Node.js

In Bun 0.1.4 the situation seems to have changed slightly. This source file let me test the shebang feature:

#!/usr/bin/env node
console.log(process.isBun);

The process.isBun variable is available to test whether Bun is running or not. According to Jarred’s comment on the version of this article appearing on Medium, this will execute using Node.js if run using Bun. But with Bun 0.1.4, it is executed by Bun:

david@nuc2:~/ws/techsparx.com$ bun -v
0.1.4
david@nuc2:~/ws/techsparx.com$ node ./shebang.js 
undefined
david@nuc2:~/ws/techsparx.com$ bun ./shebang.js 
1

This says that when explicitly run using Node.js, the variable process.isBun does not exist, because Node.js is not Bun. But, when explicitly run with Bun v0.1.4, the variable does exist, despite the shebang which says to run using Node.js.

The shebang feature has been in the Unix/Linux ecosystem since Berkeley Unix 4.2BSD. At that time, I read the 4.2BSD source code (in the mid 1980’s) and learned that the shebang is handled directly by the kernel. It does the equivalent of an exec call to invoke the interpreter named in the shebang, with the script as the stdin. The interpreter is supposed to detect and ignore the first line, and to execute whatever code it finds on stdin.

It would be incorrect for Bun to pay attention to the shebang and then execute Node.js. Thankfully that’s not what it does.

BTW, here’s an interesting and informative trick you can do with shebangs:

david@nuc2:~/ws/techsparx.com$ cat shebang.sed
#!/usr/bin/sed 1d
Hello World
david@nuc2:~/ws/techsparx.com$ chmod +x shebang.sed 
david@nuc2:~/ws/techsparx.com$ ./shebang.sed 
Hello World

The sed program has existed since the 1980’s or earlier, and is a way to perform text manipulations on streams of text. The shebang in this case says to run sed with a command to delete the first line of text. This is because sed will receive the script on stdin and by deleting the first line the output will not contain the shebang. You can use this trick to create a simple command that just prints some text, such as a list of local pizza places.

So… in Bun 0.1.4 the shebang does not cause Node.js to run the test. For Jarred to be correct, Bun 0.1.3 and earlier must have done this.

I tried to execute my code using Bun 0.1.4 and ran into a large number of compatibility issues. I have done other testing with smaller pieces of code using Bun 0.1.4, and discovered several incompatibilities. In many cases these are areas where Bun has not yet implemented a feature.

The bottom line is that at this moment Bun is incapable of executing AkashaCMS. In the following article where I say that Bun executed AkashaCMS, I was confused and was not aware of the issue where Bun respected the shebang and handed off execution to Node.js.

This means that when I ran the tests below, it was run by Node.js, which is why the resulting data shows similar processing time.

/NOTE IMPORTANT UPDATE, READ BEFORE THE REST OF THIS ARTICLE

WHAT FOLLOWS IS THE ORIGINAL ARTICLE -- BE AWARE THAT SOME DETAILS ARE INNACURATE

The Bun website ( https://bun.sh/) makes it look like a very interesting alternative for Node.js development. It is the same idea as Node.js, but promises extremely better performance. Like Node.js, Bun packages a JavaScript engine from a web browser as a server-side JavaScript platform, and it promises to implement the Node.js API's for full compatibility.

Instead of using Chrome's V8 engine, it uses the JavaScriptCore engine from Apple's Safari browser. That engine is well respected, and thought to be faster than V8, making it the first claimed advantage. Another claimed advantage is that the Bun system is written in a new language, Zig, rather than in C++ or Rust. Zig is supposed to be faster due to some technical details.

Another thing I find interesting is that both TypeScript and JSX transpilers are built-in to Bun, which should make it super easy to run TypeScript code.

The team claims 90% compatibility with Node.js packages, including using the N-API to support executing native code Node.js packages. It also uses the same node_modules infrastructure, and package lookup algorithm, of Node.js, meaning it is immediately compatible with the existing Node.js ecosystem.

By contrast, Deno (the other Node.js alternative) is incompatible with the Node.js package ecosystem, giving it a large disadvantage.

What will Node.js developers do with an alternative that's compatible with the existing ecosystem, but much faster?

The trap of simple performance tests

Already there are several videos on YouTube giving Bun a first try. Every video I've watched shows them running a few simple commands, and saying gosh wow this is so fast.

There is a well known fallacy of an overly simplistic performance test. Does running a simple script with Bun mean it is hugely faster than Node.js in real applications? That's the fallacy. To verify that Bun is indeed faster requires more in-depth testing than a few simple examples.

My idea is to try Bun with a complex case to test both compatibility and performance.

My complex test case for Bun

I have developed a static website generator (AkashaCMS) with which I've built several websites. A couple of them are pretty sizable. For example, techsparx.com has over 1600 web pages - blog posts in other words. AkashaCMS supports multiple template engines, it does server-side jQuery-like DOM manipulation, and a bunch of other stuff.

With Node.js on my laptop (a Dell Latitude E7250 w/ Core i7 and 16GB memory), rendering the website to HTML that's ready to deploy requires 30 minutes or so. If Bun lives up to its claims, that should drop to 10 minutes?

There are two scenarios of importance:

1. Can Bun execute AkashaCMS at all? Can it render everything in the techsparx.com website?
2. Can Bun render my website any faster than when using Node.js?

First observations

My first test was not to run AkashaCMS itself, but to execute a test suite associated with a Node.js package. I use Mocha to build test suites, and unfortunately there is (currently) some code incompatibilities with Bun. Some of the packages were looking for values in the process object to determine compatibility. Bun did not provide the expected values and some packages used by Mocha crashed on various compatibility tests. I reported these in their issue queue, and it seems that solutions are already underway.

Another issue is that Bun does not support a package.json dependency to packages on Github. I use Github dependencies for packages I feel don't warrant publishing to npm. The Bun issue queue already noted that problem. If you, like me, have some packages that are not published to the npm repository, but you load directly from Github, you're out of luck for now.

I did find that using npm to install the packages, then using Bun to execute code, worked flawlessly. Bun is "beta" software ... FWIW.

Rendering an AkashaCMS website using Bun

The first test scenario - can Bun render the techsparx.com website - ran flawlessly with very little in the way of issue.

This website does use a couple Github dependencies. That meant using npm install to download the dependencies.

Normally, I use scripts entries in package.json to drive the build process. This is a convenient way to record such processes, which I discussed in an earlier article: How to use npm/yarn/Node.js package.json scripts as your build tool

While Bun can directly execute package.json build scripts, these scripts run the akasharender command which is in turn a script that will execute using the node command. Instead, I ran these commands by hand:

$ bun run node_modules/akasharender/cli.js -- copy-assets config.js
$ bun run node_modules/akasharender/cli.js -- render config.js

This directly executes cli.js. The -- part was to ensure the command-line parameters were passed to akasharender rather than interpolated by Bun. The first command copies asset files to the rendered output directory, and the second renders web pages and other files into that directory.

In Node.js, the run verb is not required on the command-line, nor is the -- marker. Otherwise it is the same command-line.

The first trial passed with flying colors. It correctly ran AkashaCMS and rendered the techsparx.com website.

First Bun performance test -- copying asset files

To understand the performance figures I'll give, it may help to have a little understanding of AkashaCMS.

An AkashaCMS project has four kinds of input directories, assets, documents and two kinds of template directories. The files in the assets directories are simply copied to the rendered output directory, while files in the documents directories are run through a rendering process. The resulting output directory contains whatever mix of HTML/CSS/JS/Image files that are desired by the website author.

Therefore copy-assets is essentially these steps:

1. Look for files in the assets directories
2. Use an efficient file copy operation (fs.copy from the fs-extra package)

The file copying loop uses the fastq package to run up to 10 simultaneous copy operations. No attempt is made to skip copying files already in the output directory.

By contrast the render command runs rendering code, template engines, and a bunch more stuff.

The test system is an Intel NUC with a 5th generation Core i5, 16GB of memory, and the files stored on an HDD.

The measurements were taken using the time command. The columns below are real, meaning the elapsed time, user, meaning the CPU consumption of the user-mode code, and sys, meaning CPU consumption of kernel code.

UPDATE: NOTE THE DISCUSSION AT THE TOP /UPDATE

These are the timings for six runs of copy-assets using Bun v0.1.2.

real  0m7.326s  user  0m5.241s  sys  0m1.319s
real  0m4.445s  user  0m5.277s  sys  0m1.101s
real  0m4.847s  user  0m5.346s  sys  0m1.137s
real  0m4.874s  user  0m5.345s  sys  0m1.217s
real  0m4.847s  user  0m5.420s  sys  0m1.128s
real  0m4.867s  user  0m5.472s  sys  0m1.167s

And these are the timings for six runs of copy-assets using Node.js v18.5.0

real  0m4.686s  user  0m5.105s  sys  0m1.279s
real  0m4.549s  user  0m5.160s  sys  0m1.224s
real  0m4.659s  user  0m5.246s  sys  0m1.309s
real  0m4.884s  user  0m5.127s  sys  0m1.285s
real  0m4.658s  user  0m5.316s  sys  0m1.148s
real  0m4.968s  user  0m5.174s  sys  0m1.292s

In other words, the results are roughly the same.

For techsparx.com, there are over 2000 files to copy.

Second Bun performance test -- rendering a website full of HTML files

We've just demonstrated that Bun did not improve on Node.js in a test that is heavy on copying files. The next stage is to see how it performs with a more complex task of rendering Markdown files to HTML then processing templates in the HTML.

The primary packages used in this case are:

• Markdown rendering using markdown-it v12.x
• Server-side DOM processing using Cheerio v1.0.0-rc.10
• Template processing using a mix of EJS (v3.1.x) and Nunjucks (v3.2.x)

Like with the copy-assets command, fastq is used to execute a few renderings in parallel at a time.

UPDATE: NOTE THE DISCUSSION AT THE TOP /UPDATE

Timings for rendering techsparx.com using Bun v0.1.2

real  26m1.263s   user  25m20.973s  sys  0m27.235s
real  25m50.435s  user  25m40.301s  sys  0m24.955s
real  25m53.489s  user  25m58.259s  sys  0m25.791s

Timings for rendering techsparx.com using Node.js v18.5.0

real  25m46.665s  user  25m42.305s  sys	0m26.839s
real  26m6.209s   user  27m38.675s  sys	0m25.880s
real  25m42.731s  user  27m29.677s  sys	0m25.977s

As for the copy-assets case, it's roughly the same amount of time for both platforms.

Summary

UPDATE: NOTE THE DISCUSSION AT THE TOP /UPDATE

For this workload, Bun has roughly the same performance as Node.js.

Since the Bun team made big claims about the performance, we have to consider what's going on. Beyond, that is, recognizing that Bun is in Beta and may have a bug or two.

Obviously, this isn't the best of performance tests. Nearly 20 years ago I worked in the Java SE team at Sun Microsystems, and spent time involved with a group working on performance enhancements. They had a long list of tests of specific workloads, where each test focused on one specific performance measurement. This way the team could say that release N improved string performance by n%.

While running an application like AkashaCMS exercises a significant portion of the platform, it is not a clean focused test scenario. Just as the overly simplistic Bun demonstrations found today on YouTube don't help us understand Bun's performance, this test also does not. I see that there was a Benchmarking team within the Node.js team, but that their workspace has been idle for years. I imagine, however, that they developed the correct set of tests for the purpose.

I noticed a package - https://github.com/majimboo/node-benchmarks - that appears to have a suite of benchmark tests for execution on Node.js. But, attempting to install it using Node.js v18.5.0 failed trying to install the microtime package. Patching its package.json to use microtime v3.1.x lets the tests run, but I don't know how to interpret the results. In any case, the tests in that suite are the correct sort for this purpose.

To understand if Bun is truly faster than Node.js, and in what functional areas it is faster or slower, requires proper comparative benchmark tests. Since both platforms aim to execute the exact same code, the same benchmark/performance tests could be run on each to measure relative performance.

When Ryan Dahl first launched Node.js back in 2009, he sold the world on it by presenting performance benchmarks comparing it against other platforms. IIRC his tests focused on a useful metric, being the memory footprint and number of HTTP operations per second.

At the end of this, I'm impressed with Bun. It is a newly announced project that aims to replicate Node.js, and it's already able to handle a somewhat complex application.

UPDATE: NOTE THE DISCUSSION AT THE TOP /UPDATE