Tags: Node.JS
Following up on earlier Bun testing, as of v0.1.8, it is improving rapidly, but is still missing important functionality.
I am interested in evaluating claims that Bun is highly compatibility with Node.js while giving extremely better performance. If true, this is very attractive, since all of us would love to run our applications with higher performance.
Bun, Deno, and Node.js offer roughly the same thing, a JavaScript execution platform for use with various non-browser scenarios in server or command-line applications. Because Bun strives for Node.js compatibility, it can immediately use the Node.js package ecosystem. Deno, by contrast, cannot do so making Bun far more attractive than Deno.
The approach here is using complex applications to test two things:
- To see if Bun will execute my application(s) (AkashaCMS and related tools) at all
- Whether it will do so at higher performance than Node.js.
Testing Bun's claim with a complex application gives us real-world data about when, or if, to adopt Bun. If both conditions are met, there is no reason to not switch to Bun, right? At the moment, as we'll see shortly, it's not quite there yet.
This article was written in the time frame of August 18. Bun 0.1.8/0.1.9 was just released, and the last few releases have contained enormous improvements. The Bun team has implemented many missing features, fixed a number of segmentation faults and other issues. It is a followup to two earlier articles, the most recent being -- Deeper testing of Bun's performance and compatibility against Node.js.
In-memory database queries - but Chokidar does not work
The Chokidar is very popular, and supports dynamically watching a directory tree for changes (files added, deleted, or changed). Lots of applications use it to support automatically rebuilding things, and AkashaCMS uses it for that purpose.
AkashaCMS is a static website generator whose purpose is to take Markdown, CSS, images and other files, and produce a directory hierarchy containing HTML, CSS, and the like. It uses Chokidar to scan the input directories. It then stores data about all files in an in-memory database which is constantly queried for information while rendering the website.
Unfortunately Chokidar does not execute on Bun, because fs.watch is not implemented (
see issue 832 in Bun issue queue). The filed issue includes a simple application, and when run on Bun the error is clear - it is not supported: TypeError: fs.watch is not a function. (In 'fs.watch(path, options, handleEvent)', 'fs.watch' is undefined)
Lacking the ability to use Chokidar on Bun, I've written a pair of tests that load a YAML file containing data on a large number of files, and then run a number of queries. One test uses ForerunnerDB, and the other uses LokiJS, both of which are in-memory databases offering a MongoDB-like API. You'll find the tests in a repository at:
https://github.com/akashacms/akashacms-perftest/tree/master/bench
$ node db-forerunner.mjs
cpu: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz
runtime: node v18.6.0 (x64-linux)
benchmark time (avg) (min … max)
----------------------------------------------------------------------------
paths 82.39 ms/iter (74.77 ms … 107.53 ms)
random find 1.56 ms/iter (1.03 ms … 7.45 ms)
random siblings 1.4 ms/iter (594.58 µs … 5.93 ms)
random indexes 1.78 ms/iter (1.1 ms … 8.92 ms)
search layouts using find w/ orderBy 80.23 ms/iter (74.9 ms … 89.96 ms)
search layouts using find 77.05 ms/iter (71.96 ms … 91.36 ms)
$ bun db-forerunner.mjs
error: Cannot find package "child_process" from "/home/david/Projects/akasharender/akashacms-perftest/bench/node_modules/pem/lib/openssl.js"
Unfortunately, Bun still does not support the child_process package. This issue is blocking a large number of packages from functioning. The Bun team knows about this issue already.
Let's instead talk about the test scenarios:
paths-- queries for the pathname for every indexed filerandom find-- Selects a pathname at random, and retrieves the data for that file.random siblings-- Selects a pathname at random, and retrieves data about files in that same directory which are not the selected file.random indexes-- Selects a pathname at random, and retrieves data about files in that directory and every child directorysearch layouts-- Find all files that use a given layout template. One version sorts the list of files, and the other does not.
The same scenarios were implemented using LokiJS:
$ node db-lokijs.mjs
cpu: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz
runtime: node v18.6.0 (x64-linux)
benchmark time (avg) (min … max)
-------------------------------------------------------------
paths 1.13 ms/iter (720.57 µs … 11.22 ms)
random find 109.44 µs/iter (74.62 µs … 876.04 µs)
random siblings 92 µs/iter (42.86 µs … 5.04 ms)
random indexes 135.56 µs/iter (112.21 µs … 1.4 ms)
search layouts 281.88 µs/iter (217.32 µs … 1.45 ms)
search layouts sorted 1.33 ms/iter (870.96 µs … 2.92 ms)
$ bun db-lokijs.mjs
cpu: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz
runtime: bun 0.1.8 (x64-linux)
benchmark time (avg) (min … max)
-------------------------------------------------------------
paths 584.23 µs/iter (383.01 µs … 2.16 ms)
random find 55.57 µs/iter (35.71 µs … 794.04 µs)
random siblings 101.52 µs/iter (41.07 µs … 1.12 ms)
random indexes 122.64 µs/iter (104.22 µs … 1.34 ms)
search layouts 182.1 µs/iter (132.24 µs … 1.55 ms)
search layouts sorted 2.55 ms/iter (1.94 ms … 6.42 ms)
In this case, the same source file executes on both Node.js and Bun.
First, the LokiJS code is significantly faster than ForerunnerDB. So much faster that I've gone ahead and rewritten AkashaCMS to use LokiJS. The time to render the techsparx.com website was over 30 minutes with the ForerunnerDB implementation to around 5 minutes with LokiJS.
Secondly, notice that executing LokiJS queries on Bun is significantly faster than on Node.js. That looks good for the future prospect that Bun will deliver significantly higher performance than Node.js.
More template rendering engines work on Bun
In earlier testing, segmentation faults were triggered by several of the template engines when run under Mitata. I reported this to the Bun team, these were fixed along with a long list of other segfault errors in Bun. I'm happy to report that most of the template engines I've tested now run correctly.
$ node render-node.mjs
cpu: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz
runtime: node v18.6.0 (x64-linux)
benchmark time (avg) (min … max)
------------------------------------------------------------------
literal 136.89 ns/iter (119.73 ns … 711.75 ns)
literal list 437.11 ns/iter (381.11 ns … 1.02 µs)
ejs-list 29.48 µs/iter (24.32 µs … 474.84 µs)
ejs-list-template 3.02 µs/iter (2.78 µs … 5.39 µs)
ejs-page 86.84 µs/iter (74.43 µs … 548.41 µs)
ejs-page-template 6.59 µs/iter (6.16 µs … 7.52 µs)
handlebars-join 7.32 µs/iter (4.88 µs … 506.12 µs)
handlebars-list 6.54 µs/iter (5.06 µs … 640.28 µs)
handlebars-page 16.47 µs/iter (13.24 µs … 1.17 ms)
liquid-join 31.73 µs/iter (17.05 µs … 5.36 ms)
liquid-list 113.29 µs/iter (66.86 µs … 5.47 ms)
liquid-page 199.1 µs/iter (140.55 µs … 4.12 ms)
nunjucks-join 45.17 µs/iter (24.58 µs … 8.53 ms)
nunjucks-list 81.83 µs/iter (55.22 µs … 2.29 ms)
nunjucks-list-template 6.52 µs/iter (6.08 µs … 8 µs)
nunjucks-page 181.09 µs/iter (154.69 µs … 612.43 µs)
nunjucks-page-template 16.51 µs/iter (13.8 µs … 533.59 µs)
less-css 2.57 ms/iter (1.25 ms … 8.26 ms)
markdown-render-simple 28.44 µs/iter (18.77 µs … 2.03 ms)
markdown-render-test-suite 182.04 µs/iter (126.57 µs … 1.24 ms)
asciidoctor-render-test-suite 37.49 ms/iter (26.14 ms … 86.24 ms)
cheerio-simple 150 µs/iter (81.09 µs … 11.31 ms)
cheerio-test-suite 351.85 µs/iter (238.07 µs … 7.83 ms)
$ bun render-node.mjs
cpu: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz
runtime: bun 0.1.8 (x64-linux)
benchmark time (avg) (min … max)
------------------------------------------------------------------
literal 141.16 ns/iter (115.46 ns … 846.57 ns)
literal list 492.97 ns/iter (434.07 ns … 859.5 ns)
ejs-join 25.15 µs/iter (16.64 µs … 5.87 ms)
ejs-list 42.96 µs/iter (29.89 µs … 4.14 ms)
ejs-list-template 2.99 µs/iter (2.8 µs … 3.5 µs)
ejs-page 123.95 µs/iter (91.74 µs … 2.66 ms)
ejs-page-template 6.69 µs/iter (5.75 µs … 1.97 ms)
handlebars-join 4.13 µs/iter (2.98 µs … 2.49 ms)
handlebars-list 4.59 µs/iter (3.32 µs … 1.1 ms)
handlebars-page 10.87 µs/iter (8.14 µs … 1.08 ms)
liquid-join 36.64 µs/iter (20.21 µs … 2.48 ms)
liquid-list 132.03 µs/iter (85 µs … 1.95 ms)
liquid-page 230.89 µs/iter (179.65 µs … 2.46 ms)
nunjucks-join error: _Loader.call is not a function. (In '_Loader.call(this)', '_Loader.call' is undefined)
...
nunjucks-list error: _Loader.call is not a function. (In '_Loader.call(this)', '_Loader.call' is undefined)
...
nunjucks-list-template error: ...
...
nunjucks-page error: _Loader.call is not a function. (In '_Loader.call(this)', '_Loader.call' is undefined)
...
nunjucks-page-template error: ...
...
less-css 2.91 ms/iter (1.84 ms … 5.87 ms)
markdown-render-simple 38.14 µs/iter (22.12 µs … 2.16 ms)
markdown-render-test-suite 198.03 µs/iter (148.47 µs … 1.61 ms)
cheerio-simple 79.21 µs/iter (51.27 µs … 2.31 ms)
cheerio-test-suite 289.82 µs/iter (204.11 µs … 1.81 ms)
This demonstrates that Bun now executes every template engine (of the ones I chose to test) without triggering a segfault. Kudos to the Bun team for that. Unfortunately, the Nunjucks engine does not work on Bun. I've reported this to the Nunjucks team.
As for performance, there's a range of results. Some template engines are faster on Bun, some are slower. What stands out is that Cheerio is significantly faster on Bun. AkashaCMS uses Cheerio extensively for server-side DOM processing.
The first part of each scenario name is the template engine: literal uses JavaScript literal strings, ejs uses EJS, handlebars uses Handlebars, liquid uses LiquidJS, nunjucks uses Nunjucks, less uses LESSCSS, markdown uses Markdown-IT, asciidoctor uses AsciiDoctor, and cheerio uses Cheerio.
The join and list scenarios are relatively simple, like this:
bench('literal', () => { return `${people.join(', ')}`; });
bench('literal list', () => {
const ret = `
<ul>
${people.map(person => {
return `<li>${person}</li>`
})}
</ul>
`;
// console.log(ret);
return ret;
});
Did you know it was possible to do nested template strings in JavaScript? I didn't, but here it is.
The -page scenarios are a little more complex and is meant to mimic building a full HTML page. These require three separate templates, with two of them being rendered into the third, like so:
bench('ejs-page', () => {
const content = ejs.render(ejsContent, { people: people });
const footer = ejs.render(ejsFooter, {
branding: 'Formatted with <a href="https://example.com">ExampleCMS</a>'
});
ejs.render(ejsPage, {
title: 'Test page for performance benchmarks',
content: content,
footer: footer
});
});
How this would apply in the real world is to consider how the server-side rendering a web page is organized. It's extremely unlikely that there will be one master template to render. Instead, there will be many small snippets, with each being rendered from a template. Each page could be assembled from a dozen or maybe hundreds individual template renderings.
These results show:
- Literal strings are slower on Bun
- EJS is slower on Bun
- Handlebars is faster on Bun
- Liquid is slower on Bun
- Nunjucks doesn't execute on Bun
- LessCSS is about the same on Bun
- Markdown is slower on Bun
- AsciiDoctor doesn't execute on Bun
- Cheerio is faster on Bun
- As expected, the more complex scenarios require more compute time
While the situation is improved - Bun now executes more template engines - there aren't many template rendering performance gains here.
AsciiDoctor is similar to Markdown in that both are simple text formats for representing rich text. The AsciiDoctor package for Node.js is cross-compiled from Ruby code, and relies on the Opal package. Unfortunately the current version of AsciiDoctor/Opal fails on Bun. But, this turned up an issue in Bun where, in 0.1.8, the following code failed:
import { fileURLToPath, pathToFileURL } from 'node:url';
import * as path from 'path';
console.log(import.meta.url);
console.log(fileURLToPath(import.meta.url));
const __asciidoctorDistDir__ = path.dirname(fileURLToPath(import.meta.url))
console.log(__asciidoctorDistDir__);
This code pattern is important for emulating the __filename and __dirname variables in ES6 modules on Node.js. The issue was that the file: URL in import.meta.url was not recognized. But, a fix went in for Bun 0.1.9 and this particular code now works in Bun 0.1.9.
For EJS and Nunjucks, I also tested the gain from separately compiling the template. In other words, instead of using ejs.render or nunjucks.renderString, to use ejs.compile and nunjucks.compile to produce template objects. These are the test cases with -template in their name. From these results, precompiling the templates gives a large performance gain, but there is no significant performance difference between Node.js and Bun on precompiled templates.
Copying files (fs.copyFile) much faster with Bun
Another scenario implemented by AkashaCMS is copying files from assets directories to the rendered output directory. The task is simple, you generate a list of input files and for each file the location within the output directory. Your application then uses functions in the fs package to copy the files.
You could use fs.createReadStream and the Streams API to copy the file, but that's complex. In Node.js 14 the fs.copyFile function was added, and in Node.js 17 the fs.cp function was added. These are much simpler.
Since, Chokidar does not work on Bun (see above) we cannot read the asset directory hierarchy in the normal fashion. What I did was generate a JavaScript file containing an array of all asset files in a test website. A benchmark simple test results as so:
import * as path from 'path';
import { promises as fsp } from 'fs';
import { assets } from './assets.mjs';
import { bench, run } from "mitata";
bench('copy-assets', async () => {
for (let fileInfo of assets) {
const outdir = path.join('out', fileInfo.mountPoint);
await fsp.mkdir(outdir, { recursive: true });
const outfile = path.join('out', fileInfo.vpath);
await fsp.copyFile(fileInfo.fspath, outfile);
}
});
try {
await run({
percentiles: false
});
} catch (err) { console.error(err); }
Previously this code would have used the mkdirs and copy functions in the fs-extra package. But, the built-in functions can handle the two steps.
Another issue discovered is that Bun does not support the fs.cp function:
copy-assets error: fsp.cp is not a function. (In 'fsp.cp(fileInfo.fspath, outfile)', 'fsp.cp' is undefined)
Therefore the test uses fs.copyFile.
$ node copy-files.mjs
cpu: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz
runtime: node v18.6.0 (x64-linux)
benchmark time (avg) (min … max)
---------------------------------------------------
copy-assets 46.36 ms/iter (26.71 ms … 149.28 ms)
$ bun copy-files.mjs
cpu: Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz
runtime: bun 0.1.8 (x64-linux)
benchmark time (avg) (min … max)
---------------------------------------------------
copy-assets 10.16 ms/iter (8.43 ms … 10.88 ms)
Bun is significantly faster than Node.js for this task of copying files.
Missing packages prevent many kinds of applications
Any package containing a command-line tool which uses the Commander framework will see this error:
$ bun node_modules/akasharender/cli.js -- --help
error: Cannot find package "child_process" from "/home/david/Projects/akasharender/akashacms-perftest/node_modules/akasharender/node_modules/commander/index.js"
The Bun team knows the child_process package is missing, as well as some other packages. These packages are baked into Node.js, and should therefore be baked into Bun.
For example, the forever-agent package uses the TLS package, another baked-in package. Of course any application creating or interacting with an HTTPS service will use the TLS package. And, the cacheable-lookup package fails when loading the DNS package. Any application that deals with connecting to web services using domain names must do a DNS lookup, and will be blocked because this package is not available.
The Bun team knows about these issues, and we expect for them to be handled over time.
Summary
Bun is improving rapidly, but there are significant functionality gaps right now. Many extremely popular packages are blocked from executing on Bun because of these missing core packages or functions.
This article was written using Bun 0.1.8 and 0.1.9.
We've demonstrate that Bun is faster than Node.js in several non-trivial scenarios. In other scenarios performance between the two is roughly on par, and in some cases Node.js has the advantage. It remains to be seen how Bun will perform on a full application, because the many missing functionality pieces prevent doing so.
Bun so far is very interesting, but the missing features prevent running most complete Node.js applications on Bun. The Bun team still has a lot of work ahead of themselves.
Books by David Herron
(Sponsored)
