The Challenge of Benchmarking WebAssembly

The aforementioned suite of 24 benchmarks is the PolybenchC benchmark suite [5], which is designed to measure the effect of polyhedral loop optimizations in compilers. All the benchmarks in the suite are small scientific computing kernels rather than full applications (e.g., matrix multiplication and LU Decomposition); each is roughly 100 LOC. While WebAssembly is designed to accelerate scientific kernels on the Web, it is also explicitly designed for a much richer set of full applications.

The WebAssembly documentation highlights several intended use cases [7], including scientific kernels, image editing, video editing, image recognition, scientific visualization, simulations, programming language interpreters, virtual machines, and POSIX applications. Therefore, WebAssembly’s strong performance on the scientific kernels in PolybenchC do not imply that it will perform well given a different kind of application.

We argue that a more comprehensive evaluation of WebAssembly should rely on an established benchmark suite of large programs, such as the SPEC CPU benchmark suites. In fact, the SPEC CPU 2006 and 2017 suite of benchmarks include several applications that fall under the intended use cases of WebAssembly: eight benchmarks are scientific applications (e.g., 433.milc, 444.namd, 447.dealII, 450.soplex, and 470.lbm), two benchmarks involve image and video processing (464.h264ref and 453.povray), and all of the benchmarks are POSIX applications.

Unfortunately, it is not possible to simply compile a sophisticated native program to WebAssembly. Native programs, including the programs in the SPEC CPU suites, require operating system services, such as a filesystem, synchronous I/O, and processes, which WebAssembly and the browser do not provide. The SPEC benchmarking harness itself requires a file system, a shell, the ability to spawn processes, and other Unix facilities. To overcome these limitations when porting native applications to the web, many programmers painstakingly modify their programs to avoid or mimic missing operating system services. Modifying well-known benchmarks, such as SPEC CPU, would not only be time consuming but would also pose a serious threat to validity.

The standard approach to running these applications today is to use Emscripten, a toolchain for compiling C and C++ to WebAssembly [39]. Unfortunately, Emscripten only supports the most trivial system calls and does not scale up to large-scale applications. For example, to enable applications to use synchronous I/O, the default Emscripten MEMFS filesystem loads the entire filesystem image into memory before the program begins executing. For SPEC, these files are too large to fit into memory.

A promising alternative is to use Browsix, a framework that enables running unmodified, full-featured Unix applications in the browser [28, 29]. Browsix implements a Unix-compatible kernel in JavaScript, with full support for processes, files, pipes, blocking I/O, and other Unix features. Moreover, it includes a C/C++ compiler (based on Emscripten) that allows programs to run in the browser unmodified. The Browsix case studies include complex applications, such as LaTeX, which runs entirely in the browser without any source code modifications.

Unfortunately, Browsix is a JavaScript-only solution, since it was built before the release of WebAssembly. Moreover, Browsix suffers from high performance overhead, which would be a significant confounder while benchmarking. Using Browsix, it would be difficult to tease apart the poorly performing benchmarks from performance degradation introduced by Browsix.

Contributions

Emscripten Runtime Modifications

Browsix modifies the Emscripten compiler to allow processes (which run in WebWorkers) to communicate with the Browsix kernel (which runs on the main thread of a page). Since Browsix compiles native programs to JavaScript, this is relatively straightforward: each process’ memory is a buffer that is shared with the kernel (a SharedArrayBuffer), thus system calls can directly read and write process memory. However, this approach has two significant drawbacks. First, it precludes growing the heap on-demand; the shared memory must be sized large enough to meet the high-water-mark heap size of the application for the entire life of the process. Second, JavaScript contexts (like the main context and each web worker context) have a fixed limit on their heap sizes, which is currently approximately 2.2 GB in Google Chrome [6]. This cap imposes a serious limitation on running multiple processes: if each process reserves a 500 MB heap, Browsix would only be able to run at most four concurrent processes. A deeper problem is that WebAssembly memory cannot be shared across WebWorkers and

Browsix-Wasm uses a different approach to process-kernel communication that is also faster than the Browsix approach. Browsix-Wasm modifies the Emscripten runtime system to create an auxiliary buffer (of 64MB) for each process that is shared with the kernel, but is distinct from process memory. Since this auxiliary buffer is a SharedArrayBuffer the Browsix-Wasm process and kernel can use Atomic API for communication. When a system call references strings or buffers in the process’s heap (e.g., writev or stat), its runtime system copies data from the process memory to the shared buffer and sends a message to the kernel with locations of the copied data in auxiliary memory. Similarly, when a system call writes data to the auxiliary buffer (e.g., read), its runtime system copies the data from the shared buffer to the process memory at the memory specified. Moreover, if a system call specifies a buffer in process memory for the kernel to write to (e.g., read), the runtime allocates a corresponding buffer in auxiliary memory and passes it to the kernel. In case the system call is either reading or writing data of size more than 64MB, Browsix-Wasm divides this call into several calls such that each call only reads or writes at maximum 64MB of data. The cost of these memory copy operations is dwarfed by the overall cost of the system call invocation, which involves sending a message between process and kernel JavaScript contexts. We show in §4.2.1 that Browsix-Wasm has negligible overhead.

Performance Optimization

While building Browsix-Wasm and doing our preliminary performance evaluation, we discovered several performance issues in parts of the Browsix kernel. Left unresolved, these performance issues would be a threat to the validity of a performance comparison between WebAssembly and native code. The most serious case was in the shared filesystem component included with Browsix/Browsix-Wasm, BrowserFS. Originally, on each append operation on a file, BrowserFS would allocate a new, larger buffer, copying the previous and new contents into the new buffer. Small appends could impose substantial performance degradation. Now, whenever a buffer backing a file requires additional space, BrowserFS grows the buffer by at least 4 KB. This change alone decreased the time the 464.h264ref benchmark spent in Browsix from 25 seconds to under 1.5 seconds. We made a series of improvements that reduce overhead throughout Browsix-Wasm. Similar, if less dramatic, improvements were made to reduce the number of allocations and the amount of copying in the kernel implementation of pipes.

3.1 Browsix-SPEC Benchmark Execution

Figure 2 illustrates the key pieces of Browsix-SPEC in play when running a benchmark, such as 401.bzip2 in Chrome. First (1), the Browsix-SPEC benchmark harness launches a new browser instance using a WebBrowser automation tool, Selenium.²²2https://www.seleniumhq.org/ (2) The browser loads the page’s HTML, harness JS, and Browsix-Wasm kernel JS over HTTP from the benchmark harness. (3) The harness JS initializes the Browsix-Wasm kernel and starts a new Browsix-Wasm process executing the runspec shell script (not shown in Figure 2). runspec in turn spawns the standard specinvoke (not shown), compiled from the C sources provided in SPEC 2006. specinvoke reads the speccmds.cmd file from the Browsix-Wasm filesystem and starts 401.bzip2 with the appropriate arguments. (4) After the WebAssembly module has been instantiated but before the benchmark’s main function is invoked, the Browsix-Wasm userspace runtime does an XHR request to Browsix-SPEC to begin recording performance counter stats. (5) The benchmark harness finds the Chrome thread corresponding to the Web Worker 401.bzip2 process and attaches perf to the process. (6) At the end of the benchmark, the Browsix-Wasm userspace runtime does a final XHR to the benchmark harness to end the perf record process. When the runspec program exits (after potentially invoking the test binary several times), the harness JS POSTs (7) a tar archive of the SPEC results directory to Browsix-SPEC. After Browsix-SPEC receives the full results archive, it unpacks the results to a temporary directory and validates the output using the cmp tool provided with SPEC 2006. Finally, Browsix-SPEC kills the browser process and records the benchmark results.

4.1 PolyBenchC Benchmarks

Haas et al. [18] used PolybenchC to benchmark WebAssembly implementations because the PolybenchC benchmarks do not make system calls. As we have already argued, the PolybenchC benchmarks are small scientific kernels that are typically used to benchmark polyhedral optimization techniques, and do not represent larger applications. Nevertheless, it is still valuable for us to run PolybenchC with Browsix-Wasm, because it demonstrates that our infrastructure for system calls does not have any overhead. Figure 3(a) shows the execution time of the PolyBenchC benchmarks in Browsix-Wasm and when run natively. We are able to reproduce the majority of the results from the original WebAssembly paper [18]. We find that Browsix-Wasm imposes a very low overhead: an average of 0.2% and a maximum of 1.2%.

4.2 SPEC Benchmarks

We now evaluate Browsix-Wasm using the C/C++ benchmarks from SPEC CPU2006 and SPEC CPU2017 (the new C/C++ benchmarks and the speed benchmarks), which use system calls extensively. We exclude four data points that either do not compile to WebAssembly⁵⁵5400.perlbench, 403.gcc, 471.omnetpp, and 456.hmmer from SPEC CPU2006 do not compile with Emscripten. or allocate more memory than WebAssembly allows.⁶⁶6From SPEC CPU2017, the ref dataset of 638.imagick_s and 657.xz_s require more than 4 GB RAM. However, these benchmarks do work with their test dataset. Table 1 shows the absolute execution times of the SPEC benchmarks when running with Browsix-Wasm in both Chrome and Firefox, and when running natively.

WebAssembly performs worse than native for all benchmarks except for 429.mcf and 433.milc. In Chrome, WebAssembly’s maximum overhead is 2.5$\times$ over native and 7 out of 15 benchmarks have a running time within 1.5$\times$ of native. In Firefox, WebAssembly is within 2.08$\times$ of native and performs within 1.5$\times$ of native for 7 out of 15 benchmarks. On average, WebAssembly is 1.55$\times$ slower than native in Chrome, and 1.45$\times$ slower than native in Firefox. Table 2 shows the time required to compile the SPEC benchmarks using Clang and Chrome. (To the best of our knowledge, Firefox cannot report WebAssembly compile times.) In all cases, the compilation time is negligible compared to the execution time. However, the Clang compiler is orders of magnitude slower than the WebAssembly compiler. Finally, note that Clang compiles benchmarks from C++ source code, whereas Chrome compiles WebAssembly, which is a simpler format than C++.

Benchmark	Native	Google Chrome	Mozilla Firefox
401.bzip2	370 $\pm$ 0.6	864 $\pm$ 6.4	730 $\pm$ 1.3
429.mcf	221 $\pm$ 0.1	180 $\pm$ 0.9	184 $\pm$ 0.6
433.milc	375 $\pm$ 2.6	369 $\pm$ 0.5	378 $\pm$ 0.6
444.namd	271 $\pm$ 0.8	369 $\pm$ 9.1	373 $\pm$ 1.8
445.gobmk	352 $\pm$ 2.1	537 $\pm$ 0.8	549 $\pm$ 3.3
450.soplex	179 $\pm$ 3.7	265 $\pm$ 1.2	238 $\pm$ 0.5
453.povray	110 $\pm$ 1.9	275 $\pm$ 1.3	229 $\pm$ 1.5
458.sjeng	358 $\pm$ 1.4	602 $\pm$ 2.5	580 $\pm$ 2.0
462.libquantum	330 $\pm$ 0.8	444