Roy Triesscheijn’s Weblog

Lately I’ve been working on a rather large project (software web shop that automatically adds and updates products). The customer required that a large part of the software was automatically tested. We accomplished this by using PHPUnit (somewhat quirky, but it got the job done). However there was one big problem.

The problem

The web shop was built with the Zend Framework in true MVC fashion. This meant that it was very easy to write unit tests for the model. However the model wasn’t that complex, just a bunch of queries and some massaging of the fetched data. The most interesting things happened in the views and we had no clue how to test the views automatically. Because we had a hard deadline, and didn’t have time to fully dive into this problem, one of our testers committed himself to test all the views every day. This was feasible because we didn’t have too many views.

Current approaches

Automatically testing user interfaces is hard, especially in web languages. In a C# or Java application you can at least fill in text boxes and ‘press’ some buttons via code, although this will require you to (temporarily) make a lot of stuff public. In web languages you don’t have this amount of direct control; the closest thing would be to write a lot of Javascript. However Javascript’s state is lost on page transitions so you’d have to write the state to a file, fetch it on the next page, continue testing there, and so on. This seems like a really big bunch of work, with not much gain. However maybe there is a framework out there, which does just this thing. Maybe you can tune it to navigate the web shop for you, however there is still another big problem.

Presentation is important

For a user interface it is very important how everything looks. In web apps a small typo can make a page totally unreadable. A possible approach would be to strictly specify where a control should appear (maybe in pixels relative to the browser’s borders). But this is not possible to check with any Javascript or any other web front-end language I know. Maybe an application can make a screenshot and compare this with a screenshot that was deemed correct? Even this approach might not work very well as this would already give an error when the text on a label was partially changed.

Possible solution

I’ve come up with the following solution so far. This is just an idea which I’ve just cooked up (and the reason for this post) but I believe that it would work. Unfortunately there is no product available on the market yet that does these things.

In my opinion a good user interface tester would test in two steps. One test focused on content correctness and one focused on visual correctness.

The content correctness test would roughly follow the following steps:

• Request a page from the server

• Fill in controls and click buttons, following a predefined ‘test path’

• Verify that returned pages contains correctly filled in elements (does the label with name label1 contain the text “hello world”?). This could be done by adding a special identifier to each control that needs to be tested. The identifier can be hidden in ‘comments’ so that even during testing the web application stays compatible with normal browsers.

For every step a screenshot is made from the render output in the most popular web browsers. If the content correctness test succeeds the visual correctness test is executed, which contains the following test.

• Record control positions and sizes by ‘scanning’ the screenshots using OCR techniques. Might be aided by cross referencing the page’s source code.

• Test if controls are inside predefined ‘control locations’. Think of setting this up by dragging rectangular boxes over a screenshot to specify where a control should approximately be.

• If a control is not entirely on the correct location calculate a badness percentage.

• If the accumulated badness percentage is too high: fail the test and generate a nice picture, clearly marking the deviations.

• Also give the tester the option to select a failed test as the new correct references. This is useful for quickly updating tests when larger parts of the interface are changed.

Of course this approach doesn’t solve all the problems of testing user interfaces. And user interfaces will also have to be checked manually once in a while. However I think an application capable of performing the above tests would be very handy. Most of the above is also applicable to non-web applications btw.

I would very much like to hear everyone’s thoughts on this. How do you currently test user interfaces, have you come up with a feasible way to test large user interfaces?