Friday, May 17, 2013

Headless Testing in Dart (Take Two)

‹prev | My Chain | next›

I am pretty jazzed about the progress in both the ICE Code Editor and the test suite that is driving much of its development. I really have the sense that I have the beginnings of a robust codebase. I love Dart. One of the things that I do not have is the ability to test headlessly. This is something that worked way back in the day, but stopped at some point due to a bug in unittest.

Actually, thanks to some feedback on that bug report, it may just be a question of some additional test setup. To test that out, I first upgrade my Dart SDK and unittest. I've been stuck on:
ice-code-editor git:(master) dart --version
Dart VM version: 0.5.0.1_r21823 (Mon Apr 22 14:02:11 2013)
This keeps me back on the 0.5.0 version of unittest:
➜  ice-code-editor git:(master) ls -l packages 
total 16
lrwxrwxrwx 1 chris chris 66 May 16 00:21 browser -> /home/chris/.pub-cache/hosted/pub.dartlang.org/browser-0.5.0+1/lib
lrwxrwxrwx 1 chris chris  6 May 16 00:21 ice_code_editor -> ../lib
lrwxrwxrwx 1 chris chris 60 May 16 00:21 js -> /home/chris/.pub-cache/hosted/pub.dartlang.org/js-0.0.22/lib
lrwxrwxrwx 1 chris chris 63 May 16 00:21 meta -> /home/chris/.pub-cache/hosted/pub.dartlang.org/meta-0.5.0+1/lib
lrwxrwxrwx 1 chris chris 67 May 16 00:21 unittest -> /home/chris/.pub-cache/hosted/pub.dartlang.org/unittest-0.5.0+1/lib
After downloading the latest version of Dart, I have SDK version 0.5.7. A pub install then installs the 0.5.7 version of unittest (among others):
➜  ice-code-editor git:(master) dart --version
Dart VM version: 0.5.7.3_r22659 (Mon May 13 20:57:19 2013) on "linux_x64"
➜  ice-code-editor git:(master) pub update
Resolving dependencies...
Downloading unittest 0.5.7...
Downloading browser 0.5.7...
Downloading meta 0.5.7...
Dependencies updated!
I did not realize that it was possible to pin pub packages like this. I doubt that I will ever need to do something like that, but a quick inspection of unittest's pubspec.yaml shows that the environment property does the trick:
➜  ice-code-editor git:(master) cat ~/.pub-cache/hosted/pub.dartlang.org/unittest-0.5.7/pubspec.yaml 
name: unittest
author: "Dart Team <misc@dartlang.org>"
homepage: http://www.dartlang.org
documentation: http://api.dartlang.org/docs/pkg/unittest
description: >
 A library for writing dart unit tests.
dependencies:
  meta: any


version: 0.5.7
environment:
  sdk: ">=0.5.7"
Anyhow, now that I am on latest, I can see if things have changed with headless Dart testing. Headless testing is done with the DumpRenderTree tool that is bundled with Dart:
➜  ice-code-editor git:(master) DumpRenderTree test/index.html 
CONSOLE MESSAGE: unittest-suite-wait-for-done
Content-Type: text/plain
layer at (0,0) size 800x600
  RenderView at (0,0) size 800x600
layer at (0,0) size 800x600
  RenderBlock {HTML} at (0,0) size 800x600
    RenderBody {BODY} at (8,8) size 784x571
      RenderBlock {H1} at (0,0) size 784x37
        RenderText {#text} at (0,0) size 69x36
          text run at (0,0) width 69: "Test!"
#EOF
#EOF
Since there is no test output, it seems that the bug still exists. So I add the suggested code to the web page that provides the DOM context in which the tests run:
<html>
<head>
  <title>ICE Test Suite</title>
  <script type='text/javascript'>
    var testRunner = window.testRunner || window.layoutTestController;
    if (testRunner) {
      function handleMessage(m) {
        if (m.data == 'done') {
          testRunner.notifyDone();
        }
      }
      testRunner.waitUntilDone();
      window.addEventListener("message", handleMessage, false);
    }
    if (navigator.webkitStartDart) {
      navigator.webkitStartDart();
    }
  </script>
  <script type="application/dart" src="ice_test.dart"></script>
</head>
<body>
<h1>Test!</h1>
</body>
</html>
But, when I run this in DumpRenderTree, the output hangs with:
➜  ice-code-editor git:(master) ✗ DumpRenderTree test/index.html
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: line 54: Uncaught ReferenceError: ReceivePortSync is not defined
CONSOLE MESSAGE: Exception: The null object does not have a method 'callSync'.

NoSuchMethodError : method not found: 'callSync'
Receiver: null
Arguments: [GrowableObjectArray len:0]
CONSOLE MESSAGE: Exception: The null object does not have a method 'callSync'.

NoSuchMethodError : method not found: 'callSync'
Receiver: null
Arguments: [GrowableObjectArray len:0]
After some time, this eventually returns with:
FAIL: Timed out waiting for notifyDone to be called
FAIL: Timed out waiting for notifyDone to be called
I have seen the ReceivePortSync error before when testing js-interop. To fix, I cannot manually start the Dart engine with navigator.webkitStartDart()—I need a more complete start of the engine. So I remove the conditional above, replacing it with a <script> source that pulls in Dart's browser/dart.js:
  <script type='text/javascript'>
    var testRunner = window.testRunner || window.layoutTestController;
    if (testRunner) {
      function handleMessage(m) {
        if (m.data == 'done') {
          testRunner.notifyDone();
        }
      }
      testRunner.waitUntilDone();
      window.addEventListener("message", handleMessage, false);
    }
    // if (navigator.webkitStartDart) {
    //   navigator.webkitStartDart();
    // }
  </script>
  <script src="packages/browser/dart.js"></script>
  <script type="application/dart" src="ice_test.dart"></script>
With that, I do see my test output:
➜  ice-code-editor git:(master) ✗ DumpRenderTree test/index.html
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: PASS: defaults defaults to auto-update the preview
CONSOLE MESSAGE: PASS: defaults starts an ACE instance
CONSOLE MESSAGE: PASS: defaults defaults to disable edit-only mode
CONSOLE MESSAGE: PASS: content can set the content
...
CONSOLE MESSAGE:
CONSOLE MESSAGE: All 29 tests passed.
CONSOLE MESSAGE: unittest-suite-success
Unfortunately, it still hangs at this point. Eventually, I get notifyDone errors:
FAIL: Timed out waiting for notifyDone to be called
FAIL: Timed out waiting for notifyDone to be called
Taking a closer look at the supplied runner code, I see that, to notify the test runner that it is done, a message with the contents "done" needs to be posted:
      function handleMessage(m) {
        if (m.data == 'done') {
          testRunner.notifyDone();
        }
      }
I am unsure if the unittest library is supposed to post that message. Regardless of whether it is supposed to send the message, it is not. So it seems that I need to add a way to poll for the test cases to be complete (there are no tests-done Futures in unittest).

To poll, I import the dart:async library for access to the Timer class. The pollForDone() function can then be added to my test suite as:
library ice_test;

import 'package:unittest/unittest.dart';
import 'dart:html';
import 'dart:async';
import 'package:ice_code_editor/ice.dart';
// ...
main(){
  // Run tests here...
  pollForDone(testCases);
}

pollForDone(List tests) {
  if (tests.every((t)=> t.isComplete)) {
    window.postMessage('done', window.location.href);
    return;
  }

  var wait = new Duration(milliseconds: 100);
  new Timer(wait, ()=> pollForDone(tests));
}
Given the list of all test cases, this checks if every one of them is complete. If they are, then it posts the 'done' message for the test runner and exits. If some tests are incomplete, then it waits for 100 milliseconds before trying again.

And that does the trick! The test suite still runs in Dartium and now it runs headless as well:
➜  ice-code-editor git:(master) ✗ time DumpRenderTree test/index.html
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: PASS: defaults defaults to auto-update the preview
CONSOLE MESSAGE: PASS: defaults starts an ACE instance
CONSOLE MESSAGE: PASS: defaults defaults to disable edit-only mode
CONSOLE MESSAGE: PASS: content can set the content
...
CONSOLE MESSAGE: PASS: sharing menu should close when share dialog activates

CONSOLE MESSAGE:
CONSOLE MESSAGE: All 29 tests passed.
CONSOLE MESSAGE: unittest-suite-success
...
DumpRenderTree test/index.html  0.77s user 0.10s system 100% cpu 0.870 total
The pollForDone() function is a bit of a hassle, but one that I am happy to live with if it gets me headless testing. Still, I will post that solution back onto the bug to see if there is some way that unittest could post the 'done' message itself.


Day #754

No comments:

Post a Comment