Abusing chromium headless as a test runner

You know how one of the biggest problems with the js ecosystem is that there's not enough frontend frameworks? Yeah, I decided to fix that and write my own 1. Before getting too far I already got stuck in another rabbit hole. I wanted something to run my tests automatically - refreshing manually got old fast. That's a solved problem (karma, mocha, selenium… whatever), but why not solve it again?

Enter chromium headless, which has a repl!

niklas@fasching:~/Documents/chocolate.js$ chromium-browser --headless --repl
>>> 1 + 2
{"result":{"description":"3","type":"number","value":3}}
>>>

The idea:

  1. Run chromium headless

  2. In the repl: Import module containing tests

  3. In the test framework: Once finished put test results in some property on window

  4. In the repl: Poll that window property in a loop

  5. In the bash script: grep repl stdout and print/exit/… once we found the results

And yup, that works. But not without jumping through a few hoops:

I ended up something similar to the following:

#!/usr/bin/env bash

if [[ $# -eq 0 ]] ; then
    echo "USAGE: $0 FILE PORT TIMEOUT_SECONDS"
    exit 0
fi

file=$1
port=${2:-8000}
timeout=${3:-10}

(python -m SimpleHTTPServer $port &> /dev/null &)
server_pid=$!

result=$({ echo "import('http://localhost:$port/$file').then(() => {}, (err) => window.err = err.message);" ;
           for i in $(seq 1 $(bc <<< "$timeout / 0.1")); do echo "(window.Test.results || window.err)"; sleep 0.1; done } | \
             chromium-browser --headless --repl http://localhost:$port/just-for-the-origin 2> /dev/null | (
             grep '"type":"string"' -m1
             pkill -f " --headless --repl"))

if [ $? -eq 1 ]; then
    echo "timeout (${timeout}s) waiting for test results"
    exit 1
fi

kill -9 $server_pid 2> /dev/null

echo -e "${result:36:-4}" # extract value from ">>> {"result":{"type":"string","value":"..."}}"
if echo -e "${result:36:-4}" | grep -P "^not ok" > /dev/null; then exit 1; else exit 0; fi

This isn't too pretty and the polling means we're not getting live test output but only at the end. Also my test harness has to expose the output on window to allow us to poll for it at all… So yeah, it's not all sunshine and rainbows.

Nevertheless, it works well enough for my purposes and I'm happy. Here's some example output:

import {test} from "../src/test.mjs";

test("foo", t => {
  t.equal(1, 2);
});
./etc/runner.sh  test/index.mjs 8000 1
not ok 1 foo
# - actual: 1
# - expected: 2
# - op: equal
1..1

echo $?
1

time ./etc/runner.sh  test/index.mjs 8000 10
# [...]

real	0m0,745s
user	0m0,042s
sys	0m0,005s

For now it works well enough - but maybe things will break down if output get's too long or something. Let's see.

Footnotes


1

I mean… I wanted to build a web app and got tired of manipulating the dom manually. And I didn't want to use a library / framework because that's the perfect opportunity to do a deep dive on how those work.

I got into angular pretty soon after I started programming and boy did it fuck with my brain. So much black magic. I haven't done much frontend stuff since and want to get into that again - so yeah… Prepare for yet another framework. I'm sorry world.

2

For more info, check e.g. this stackoverflow question. Running a server is not that much hassle - we're probably already doing that for dev anyways.

3

https://unix.stackexchange.com/questions/366797/grep-slow-to-exit-after-finding-match - so that's why pipes sometimes seem to get "stuck" - never thought about how things work. TIL.