HTMLUnit!

First of all, I just wanna say : You guys at HTMLUnit rock!

HTMLunit is a framework written in Java that simulates a client application to a your webserver, i.e. a browser. This one does that by going to a site, parsing through all the site and executing all javascript it comes across. And it works to! We use a lot of ajax on our site, and everything seems to work. It even handles all the jQuery stuff very well. The guys who developed this framework must really have put a lot of work in reproduction all of them browser bugs out there that jQuery ,among other, tries to solve. Respect.

That beeing said, lets complain a little. But not about htmlunit, but about Java. That damn language is so verbose. I want to execute some javascript and get the result to run my tests. No big deal, but in Java i have to type something like :
(new WebClient (
new URL("http://www.google.com"))).
getPage().executeJavaScriptResult("1+1").
getScriptResult()
where as in Ruby, this would probably be something like:
Web::Client::Page.
new("http://google.com").
js("1+1").result
This ain't a problem if you sit on eclipse, IntelliJ or something. But the thing is, I don't. And then you have to wrap all of it in classes, main etc. I started doing this but then I quickly converted everything to Jruby, or rather I just started programming in ruby instead and let jruby handle the gap between the lanugages.

The blog Scraping Dynamic Websites Using JRuby and HtmlUnit helped me getting started. But I didn't get the tarball there to work. Instead I downloaded the newest version of HMTLUnit from sourgeforge. After a couple of minutes I was up and running.

It first I tried to just use HTMLunit as a javascript runner, and use JSSpec as the testframework. But a) I didn't get it to function like I expected/ wanted to. b) There are some serious state-charing-related issues that has to be solved. So therefore I just hooked it up with rspec under jruby. It gave me a java-horror-stacktrace but then it just worked!

I was running javascript tests from the console, and it was some pretty advanced stuff. 'Did that element fadeout?, was that element updated? Did I get that response from the server'. It worked, but I wanna give a warning about how slow it was. It was about 1 sec / test. So if you have 1100 tests as we have on the backend....

Now it's just to hooked it all up with CI and we're ready to go.

I'll soon be back with a hands-on tutorial on how to set ut everything. 'til then. Bye!

Fixin', Hackin' , Patchin' & givin' up...

I've been trying to get the Crosscheck-framework now. I've solved the problem mentioned in the last post by patching jQuery. It turned out the problem was the how Crosscheck enumerates js-objects. Anyways, once that problem was solved a new one emerged. I was planning on submitting all the fixes for these problems here, but now I reached the point were giving up was the only option. There is no use in trying to get the framework to play with the jQuery-library; the differences between crosscheck and the real browser is to big. Sadly enough.

So what to do next? Either we have to go over to in-browser-testing and in that case I figure we'll go with JSSpec, which is a behavior-driven javascript testing framework. But that is the last resort; if we do our tests in the browser the CI-process is practically impossible. Which means we'll end up not running the javascript-tests, which is kind of equal to not testing at all. The other possiblity is to look at other 'browser simulators'.

John Resig, one of the most quoted javascript gurus here at the office (the other one being Douglas Crockford ), presented an interesting alternative in his blogpost Bringing the Browser to the Server. In this post he supplies a simple browser enviroment to be used together with Rhino (Mozilla's server side javascript runtime). This could then be used for javascript testing. I tried it out, but as he claims in the post, it only supplies the very basic and therefore some important things are missing. I really have to pay my respects to Resig though, the existing work is real nice and if he decides to continuing the work on it, it'll definitely be worth checking out.

So for now we can't go with crosscheck and we can't go with resig's solution. Unfortunately. Next thing to check out is the HtmlUnit framework. It has some SoundCloud specific drawbacks, one of them being that it's written in java, but on the flipside is that their developers have tried it with different versions of jQuery and it seeems to work. So expect to hear more about HtmlUnit from me.

The Hells of Crosscheck

I know you're never supposed to refer to the title in the body of your text but now I do it anyways. The hell isn't really the crosscheck framework, but the act of faking a browser. There should be no surprises that this eventually will lead to problems, the question is rather how important these are. In my case these problems seems to be quite big : I can't seem to model the events correctly. Lets tell the story from the beginning.

I decided to integrate the javascript testing with our existing testing process, meaning setting up tests that are run on each 'build', i.e. each time a developer commits code to the main repository. Crosscheck was the framework I used to achieve this, and because it is written in java and we're going with ruby on rails, it meant setting up tests and running test in java from ruby. That was really no big issue and after a couple of hour I had it all set up. I even hocked it up with jslint, so that all the javascript-files runes through that each commit. It worked super, but the problems started to come soon after I tried to use it in the development process. I of course tested this before, but there is one thing I didn't test; event handling.

Sooner or later when you test the frontend layer of an (web)application you have to test the event handling, because in many cases the events is the main interface, the entry point to the code. Not to test that the events are fired like expected would be absurd. It therefore was a major letdown to learn that crosscheck doesn't play well with jQuery's event handling. For some reason the type of the event object doesn't seem to be set in a proper way, i.e. event.type has the value undefined.

This seems like an obvious bug in the crosscheck framework but after a couple of hours diving into the source code (*phew*) it seems like everything is set up correctly, so only one question remains: why doesn't it work? Before this is solved, we can't use crosscheck to any extend. No other framework seems to do it either, because we really want the CI-integration feature. If the tests don't run on each commit, they're not going to be run. That is at least what we think...

I definitely is going to post when I found a solution/work-around/other for this problem. But right now I've reached a dead end.

Manual Testing

So where back from christmas and just got a new release up and running. I've been talking with some more experinced QA-people and also been reading some articles on it. They all use some sort of scripts to test their things. A also read that you have to specify what your system should be able to do to effectively do automated tests of it. This and the fact that I thought our testing procedure was a bit unorganized was my main inspiration to make the whole team do some manual testing before this released.

I've compiled a numbers of script like tests that I've put in an Excel sheet. Each tab is divided into different categories, testing different areas of our site. We then sort of divided those tabs between each other and went testing like h*ll. These tests looks like this:
No
1
Description
Successfull log in

Page (if any)
/login

Prerequisites
User goes to login page, types correct password

Expected Behaviour
User is redirected to dashboard

The first thing we came across was how-boring-it-was. A couple of them are ok, but after a while it becomes so slow. So slow. One of the main ideas is of course to automate as much of this as possible. But not everything can be automated, and as you do manual testing you tend to test a whole lot more than just the thing you set out to test. The thing is that humans very soon starts to improvise when they test and therefore catch more bugs. We also learned that you spot a lot of this that you weren't supposed to test, but got tested on the way.

Everybody agreed that this style of testing was way superior and that we caught a lot of bugs we definitely would have missed. But I got to warn about how time consuming this was. Me for example went though some tests, discovered some other, mostly related, bug. Fixed that, submitted the patch and so on. These were almost all small fixes but all in all it took a long time. Better would maybe had been to had noted and reported some bugs but let them slip this release.

Just a note. You can't do without manual testing. You really can't and this more organized style of testing is needed.

Meeting with the Ableton CTO.

Our team had the opportunity to meet the CTO of Ableton and get some general advices. He was of course a real clever guy who knows a great deal about developing software. We got to discuss testing and quality with him and they have it all: testdepartment, testframeworks, inhouse developed acceptence tests, etc. They also had a small group of dedicated users that received prereleases to evalutate. This proved to be a valuable things, not only for discovering bugs and see what needs to be changed, but also because they tend to hype the next version of Abletons software.

Something intresting that came up was that before they had a separate testdepartment they had this one guy who always managed to find the most bugs in their software. He applied the infamous 'monkey test strategy', i.e. randomly klick on everything as much as possible. Surprisingly this, as I said, proved to be the most successful way of finding bugs and he thought it could be a good idea to program some kind of 'bot' who randomly presses everything. But if I should do that I have to find a way of recording what the bot done, otherwise it would just come up with a bug and left us with no clue how to reproduce it.

Lets stop and think about this. This would be really intresting to try out. The way I imaging doing this is to have a couple of conditions that always has to be satisfied and then let the bot randomly visit the site and check that these conditions are met. One example of such a conditions is the accessability of tracks. We have provide the ablitiy to upload and share tracks on our site and the user can choose which persons to share their tracks with. This is of course quite errorprone and very bad for our reputation if we fail to keep the private tracks private.

The next problem is to record what has happend, Ableton has solved this by secretly recording every event in their program. If the programs then crashes they could just replay the events to reproduce the bugs and see what went wrong. Nowadays this is very helpful because they can let the users send them their crashing program and just replay it to find the bug.

Another part of quality, that we discussed, is planing. According to the Ableton CTO one sure way of introducing bugs is to plan to much features to the release. His general idea about this was that it's better to have very few really good features that works very well than to have very many features that work ok. This could seem obvious, but in reality it's not. There are so many things you like to cram in there and everybody knows that it hurts to 'kill your darlings'. Then comes the thing that people tend to underestimate how long time things take to develop. According to the CTO only 10% of software projects are finished in time, and I think none are finished with all feautures that were planed for it.

Oh and another tip we got was: merge often, release often.

A quick evaluation of testing frameworks : htmlunit

I had a quick look at htmlunit. It's a kind of a browser emulator, it parses html sites for you and even has the abliity to run some javascript. Then you could 'click links', 'fill in fields' and 'press buttons' on your website by calling some functions on the emulator.

The framework, which is written i java, seems kind of "oldschool" and doesn't seem to handle all the ajax we're doing on our site.It has some support for (XMLHTTPRequest) but that is kind of it. One nice thing about it is that it doesn't provide testingmethods (such as assertions) so you have to use an external testingframework to do this. This lets you use the same framework for all of you application and homogenise your tests. Nice.

Because we're using ruby on rails to develop our site and htmlunit is written in java we could have some trouble integrating it and probably have to use JRuby to get it to function. This is doable but think about it, first were using JRuby (a ruby runtime implemented in java) instead of ruby, this could be the cause of numerous bugs. Then on top of this we should use a browser emulator instead of a real browser, which will introduce even more bugs and glitches that, in reality, isn't there. The problem with this framework for us seems to be that we could spend to much time 'chasing ghosts'; even if the tests fails that doesn't mean that we have a bug. This is of course always a problem with testing, but this seems even worse.

Instead of this framework I stumbled over the Hpricot_forms framework. Hpricot is a nice framework for parsing html/xml content and the hpricot_forms is kind of a htmlunit port for ruby. It's really nothing fancy but it lets you "click links" and "fill in and submit forms" in your tests. I downloaded the project and after some troubles, including manually updating some code, I had it running and wrote some tests in it! It's nice and really supply a valuable addition to our testframework Rspec, but I think it would be even better if combined with the Storyrunner included in the next Rspec release.

A Quick Evaluation of Testing Frameworks : Selenium

So it seems that everybody that's coding rails and ajax stuff is talking about Selenium. So what is all the fuzz about?. Selenium is a acceptence testing framework, i.e. a tools that you use to test the final product.

The idea is to tests things like : if I click on the inbox button my messages should show. When I'm on the inbox page I should be able to click on the 'mark as read' - button and the message should be marked as read.

To people doin' agile development this may seem familiar; it resembles 'user stories'.

The tests seems easy to write and Selenium seems to be easy to integrate with rails, there is even a selenium-on-rails project. I've heard some people complain about it, saying that it is to much work and that the tests run to slow. I just had a quick look at it but to me it seems to be a good alternative. I mean, this tool runs in almost any browser and clicks the buttons for you. What is the alternative? To manually click on every single button/link/thing in every supported browser. That seems very tedious...

Any ways, the benifit of the this tool is the what-you-see-is-what-you-get feature. As with any browser-boot, it actually hijacks your browser and surf the site for you. It the site works for selenium it works for you. The major drawback seems to be speed. I wrote 3 simple tests and then ran them 200 times, equals 600 tests. This took me 11 minutes! Compare this to the 500 backend test we have, which run in about 30 seconds... We could easily have around 1500 tests in a year and then it takes 30 minutes to run them. This really is a drawback. One other thing is that it requires a GUI to run, which makes it hard to integrate with the continuous integration system.

After discussing the issue in our company it seems that the benifits seems to be greater than the drawbacks. We just bought a dedicated testing machine and hooked it up with a couple of operating systems and different browsers. The idea would be to have selenium running there now and then. And when it comes to speed : what are the alternatives? We have to test the application.

Considering the 'user story' nature of selnium it would seem like an ideal idea to integrate it with the Rbehave framework I described before. And after surfing the web for a while it turns out that this has actually been done. It still seems a bit early and 'cutting-edge' but the result is nice (taken from Kerry Buckley):

Story: Login
The front page should contain a login form

Scenario: Loading the home page when not logged in

When the user goes to /
Then the title should be 'mojo'
And the page should contain the text 'Welcome to mojo'
And the page should contain the text 'Sign in using your OpenID'
And there should be a field named 'openid_url'
And there should be a submit button named 'login', with the label 'Sign In'

steps_for(:selenium) do
When "the user goes to $path" do |path|
$selenium.open path
end
When "the user types '$text' into the $field field" do |text, field|
$selenium.type field, text
end
When "the user clicks the $button button" do |button|
$selenium.click button
$selenium.wait_for_page_to_load 5000
end
Then "the title should be '$title'" do |title|
$selenium.title.should == title
end
Then "the page should contain the text '$text'" do |text|
$selenium.should have_text_present(text)
end
Then "there should be a field named '$field'" do |field|
$selenium.should have_element_present(field)
end
Then "there should be a submit button named '$name', with the label '$label'" do |name, label|
$selenium.should have_element_present("//input[@type='submit'][@name='#{name}'][@value='#{label}']")
end
end