HTMLUnit!

First of all, I just wanna say : You guys at HTMLUnit rock!

HTMLunit is a framework written in Java that simulates a client application to a your webserver, i.e. a browser. This one does that by going to a site, parsing through all the site and executing all javascript it comes across. And it works to! We use a lot of ajax on our site, and everything seems to work. It even handles all the jQuery stuff very well. The guys who developed this framework must really have put a lot of work in reproduction all of them browser bugs out there that jQuery ,among other, tries to solve. Respect.

That beeing said, lets complain a little. But not about htmlunit, but about Java. That damn language is so verbose. I want to execute some javascript and get the result to run my tests. No big deal, but in Java i have to type something like :
(new WebClient (
new URL("http://www.google.com"))).
getPage().executeJavaScriptResult("1+1").
getScriptResult()
where as in Ruby, this would probably be something like:
Web::Client::Page.
new("http://google.com").
js("1+1").result
This ain't a problem if you sit on eclipse, IntelliJ or something. But the thing is, I don't. And then you have to wrap all of it in classes, main etc. I started doing this but then I quickly converted everything to Jruby, or rather I just started programming in ruby instead and let jruby handle the gap between the lanugages.

The blog Scraping Dynamic Websites Using JRuby and HtmlUnit helped me getting started. But I didn't get the tarball there to work. Instead I downloaded the newest version of HMTLUnit from sourgeforge. After a couple of minutes I was up and running.

It first I tried to just use HTMLunit as a javascript runner, and use JSSpec as the testframework. But a) I didn't get it to function like I expected/ wanted to. b) There are some serious state-charing-related issues that has to be solved. So therefore I just hooked it up with rspec under jruby. It gave me a java-horror-stacktrace but then it just worked!

I was running javascript tests from the console, and it was some pretty advanced stuff. 'Did that element fadeout?, was that element updated? Did I get that response from the server'. It worked, but I wanna give a warning about how slow it was. It was about 1 sec / test. So if you have 1100 tests as we have on the backend....

Now it's just to hooked it all up with CI and we're ready to go.

I'll soon be back with a hands-on tutorial on how to set ut everything. 'til then. Bye!

Fixin', Hackin' , Patchin' & givin' up...

I've been trying to get the Crosscheck-framework now. I've solved the problem mentioned in the last post by patching jQuery. It turned out the problem was the how Crosscheck enumerates js-objects. Anyways, once that problem was solved a new one emerged. I was planning on submitting all the fixes for these problems here, but now I reached the point were giving up was the only option. There is no use in trying to get the framework to play with the jQuery-library; the differences between crosscheck and the real browser is to big. Sadly enough.

So what to do next? Either we have to go over to in-browser-testing and in that case I figure we'll go with JSSpec, which is a behavior-driven javascript testing framework. But that is the last resort; if we do our tests in the browser the CI-process is practically impossible. Which means we'll end up not running the javascript-tests, which is kind of equal to not testing at all. The other possiblity is to look at other 'browser simulators'.

John Resig, one of the most quoted javascript gurus here at the office (the other one being Douglas Crockford ), presented an interesting alternative in his blogpost Bringing the Browser to the Server. In this post he supplies a simple browser enviroment to be used together with Rhino (Mozilla's server side javascript runtime). This could then be used for javascript testing. I tried it out, but as he claims in the post, it only supplies the very basic and therefore some important things are missing. I really have to pay my respects to Resig though, the existing work is real nice and if he decides to continuing the work on it, it'll definitely be worth checking out.

So for now we can't go with crosscheck and we can't go with resig's solution. Unfortunately. Next thing to check out is the HtmlUnit framework. It has some SoundCloud specific drawbacks, one of them being that it's written in java, but on the flipside is that their developers have tried it with different versions of jQuery and it seeems to work. So expect to hear more about HtmlUnit from me.

The Hells of Crosscheck

I know you're never supposed to refer to the title in the body of your text but now I do it anyways. The hell isn't really the crosscheck framework, but the act of faking a browser. There should be no surprises that this eventually will lead to problems, the question is rather how important these are. In my case these problems seems to be quite big : I can't seem to model the events correctly. Lets tell the story from the beginning.

I decided to integrate the javascript testing with our existing testing process, meaning setting up tests that are run on each 'build', i.e. each time a developer commits code to the main repository. Crosscheck was the framework I used to achieve this, and because it is written in java and we're going with ruby on rails, it meant setting up tests and running test in java from ruby. That was really no big issue and after a couple of hour I had it all set up. I even hocked it up with jslint, so that all the javascript-files runes through that each commit. It worked super, but the problems started to come soon after I tried to use it in the development process. I of course tested this before, but there is one thing I didn't test; event handling.

Sooner or later when you test the frontend layer of an (web)application you have to test the event handling, because in many cases the events is the main interface, the entry point to the code. Not to test that the events are fired like expected would be absurd. It therefore was a major letdown to learn that crosscheck doesn't play well with jQuery's event handling. For some reason the type of the event object doesn't seem to be set in a proper way, i.e. event.type has the value undefined.

This seems like an obvious bug in the crosscheck framework but after a couple of hours diving into the source code (*phew*) it seems like everything is set up correctly, so only one question remains: why doesn't it work? Before this is solved, we can't use crosscheck to any extend. No other framework seems to do it either, because we really want the CI-integration feature. If the tests don't run on each commit, they're not going to be run. That is at least what we think...

I definitely is going to post when I found a solution/work-around/other for this problem. But right now I've reached a dead end.

Manual Testing

So where back from christmas and just got a new release up and running. I've been talking with some more experinced QA-people and also been reading some articles on it. They all use some sort of scripts to test their things. A also read that you have to specify what your system should be able to do to effectively do automated tests of it. This and the fact that I thought our testing procedure was a bit unorganized was my main inspiration to make the whole team do some manual testing before this released.

I've compiled a numbers of script like tests that I've put in an Excel sheet. Each tab is divided into different categories, testing different areas of our site. We then sort of divided those tabs between each other and went testing like h*ll. These tests looks like this:
No
1
Description
Successfull log in

Page (if any)
/login

Prerequisites
User goes to login page, types correct password

Expected Behaviour
User is redirected to dashboard

The first thing we came across was how-boring-it-was. A couple of them are ok, but after a while it becomes so slow. So slow. One of the main ideas is of course to automate as much of this as possible. But not everything can be automated, and as you do manual testing you tend to test a whole lot more than just the thing you set out to test. The thing is that humans very soon starts to improvise when they test and therefore catch more bugs. We also learned that you spot a lot of this that you weren't supposed to test, but got tested on the way.

Everybody agreed that this style of testing was way superior and that we caught a lot of bugs we definitely would have missed. But I got to warn about how time consuming this was. Me for example went though some tests, discovered some other, mostly related, bug. Fixed that, submitted the patch and so on. These were almost all small fixes but all in all it took a long time. Better would maybe had been to had noted and reported some bugs but let them slip this release.

Just a note. You can't do without manual testing. You really can't and this more organized style of testing is needed.