Abusing chromium headless as a test runner

• js
• hacky

You know how one of the biggest problems with the js ecosystem is that there's not enough frontend frameworks? Yeah, I decided to fix that and write my own ¹. Before getting too far I already got stuck in another rabbit hole. I wanted something to run my tests automatically - refreshing manually got old fast. That's a solved problem (karma, mocha, selenium… whatever), but why not solve it again?

UPDATE [30.08.20]: In the meantime I switched to a go version that doesn't use the repl but the devtools protocol console api for communication: https://github.com/niklasfasching/goheadless

• More robust against errors
• But also 10x the lines of code (~30 -> ~300 LOC)

Enter chromium headless, which has a repl!

niklas@fasching:~/Documents/chocolate.js$ chromium-browser --headless --repl
>>> 1 + 2
{"result":{"description":"3","type":"number","value":3}}
>>>

The idea:

1. Run chromium headless
2. In the repl: Import module containing tests
3. In the test framework: Once finished put test results in some property on window
4. In the repl: Poll that window property in a loop
5. In the bash script: grep repl stdout and print/exit/… once we found the results

And yup, that works. But not without jumping through a few hoops:

• We can't use the file protocol - we need a server.
  At first I thought we could get away with using the file protocol + the --allow-file-access-from-files flag. While that solves CORS, we still can't use it with modules because the MIME type check fails with non-JavaScript MIME type of "" ².

• chromium repl expects a terminal and doesn't like being piped into grep.
  Polling the repl is simple as we can just pipe output of a for loop into stdin. When we pipe the output into grep though nothing happens.

  echo 1 + 2 | chromium-browser --headless --repl
  >>> {"result":{"description":"3","type":"number","value":3}}
  echo 1 + 2 | chromium-browser --headless --repl | grep
  # nothing

  strace (echo 1 + 2 | strace -e write -s1000 chromium-browser --headless --repl | grep value) confirms that there's just no write. The repl just doesn't like being piped - doesn't matter, as there's ways of pretending stdout is a tty. This also takes care of correctly handling EPIPE broken from grep -m1 so we get the poll results before the timeout / for-loop ends. EDIT: somehow the repl is ok with piping now. changed to just pkilling chromium myself - apparently it still ignores the broken pipe ³.

I ended up something similar to the following:

#!/usr/bin/env bash

if [[ $# -eq 0 ]] ; then
    echo "USAGE: $0 FILE PORT TIMEOUT_SECONDS"
    exit 0
fi

file=$1
port=${2:-8000}
timeout=${3:-10}

(python -m SimpleHTTPServer $port &> /dev/null &)
server_pid=$!

result=$({ echo "import('http://localhost:$port/$file').then(() => {}, (err) => window.err = err.message);" ;
           for i in $(seq 1 $(bc <<< "$timeout / 0.1")); do echo "(window.Test.results || window.err)"; sleep 0.1; done } | \
             chromium-browser --headless --repl http://localhost:$port/just-for-the-origin 2> /dev/null | (
             grep '"type":"string"' -m1
             pkill -f " --headless --repl"))

if [ $? -eq 1 ]; then
    echo "timeout (${timeout}s) waiting for test results"
    exit 1
fi

kill -9 $server_pid 2> /dev/null

echo -e "${result:36:-4}" # extract value from ">>> {"result":{"type":"string","value":"..."}}"
if echo -e "${result:36:-4}" | grep -P "^not ok" > /dev/null; then exit 1; else exit 0; fi

This isn't too pretty and the polling means we're not getting live test output but only at the end. Also my test harness has to expose the output on window to allow us to poll for it at all… So yeah, it's not all sunshine and rainbows.

Nevertheless, it works well enough for my purposes and I'm happy. Here's some example output:

import {test} from "../src/test.mjs";

test("foo", t => {
  t.equal(1, 2);
});

./etc/runner.sh  test/index.mjs 8000 1
not ok 1 foo
# - actual: 1
# - expected: 2
# - op: equal
1..1

echo $?
1

time ./etc/runner.sh  test/index.mjs 8000 10
# [...]

real	0m0,745s
user	0m0,042s
sys	0m0,005s

For now it works well enough - but maybe things will break down if output get's too long or something. Let's see.