A Quick Evaluation of Testing Frameworks : Selenium

So it seems that everybody that's coding rails and ajax stuff is talking about Selenium. So what is all the fuzz about?. Selenium is a acceptence testing framework, i.e. a tools that you use to test the final product.

The idea is to tests things like : if I click on the inbox button my messages should show. When I'm on the inbox page I should be able to click on the 'mark as read' - button and the message should be marked as read.

To people doin' agile development this may seem familiar; it resembles 'user stories'.

The tests seems easy to write and Selenium seems to be easy to integrate with rails, there is even a selenium-on-rails project. I've heard some people complain about it, saying that it is to much work and that the tests run to slow. I just had a quick look at it but to me it seems to be a good alternative. I mean, this tool runs in almost any browser and clicks the buttons for you. What is the alternative? To manually click on every single button/link/thing in every supported browser. That seems very tedious...

Any ways, the benifit of the this tool is the what-you-see-is-what-you-get feature. As with any browser-boot, it actually hijacks your browser and surf the site for you. It the site works for selenium it works for you. The major drawback seems to be speed. I wrote 3 simple tests and then ran them 200 times, equals 600 tests. This took me 11 minutes! Compare this to the 500 backend test we have, which run in about 30 seconds... We could easily have around 1500 tests in a year and then it takes 30 minutes to run them. This really is a drawback. One other thing is that it requires a GUI to run, which makes it hard to integrate with the continuous integration system.

After discussing the issue in our company it seems that the benifits seems to be greater than the drawbacks. We just bought a dedicated testing machine and hooked it up with a couple of operating systems and different browsers. The idea would be to have selenium running there now and then. And when it comes to speed : what are the alternatives? We have to test the application.

Considering the 'user story' nature of selnium it would seem like an ideal idea to integrate it with the Rbehave framework I described before. And after surfing the web for a while it turns out that this has actually been done. It still seems a bit early and 'cutting-edge' but the result is nice (taken from Kerry Buckley):

Story: Login
The front page should contain a login form

Scenario: Loading the home page when not logged in

When the user goes to /
Then the title should be 'mojo'
And the page should contain the text 'Welcome to mojo'
And the page should contain the text 'Sign in using your OpenID'
And there should be a field named 'openid_url'
And there should be a submit button named 'login', with the label 'Sign In'

steps_for(:selenium) do
When "the user goes to $path" do |path|
$selenium.open path
end
When "the user types '$text' into the $field field" do |text, field|
$selenium.type field, text
end
When "the user clicks the $button button" do |button|
$selenium.click button
$selenium.wait_for_page_to_load 5000
end
Then "the title should be '$title'" do |title|
$selenium.title.should == title
end
Then "the page should contain the text '$text'" do |text|
$selenium.should have_text_present(text)
end
Then "there should be a field named '$field'" do |field|
$selenium.should have_element_present(field)
end
Then "there should be a submit button named '$name', with the label '$label'" do |name, label|
$selenium.should have_element_present("//input[@type='submit'][@name='#{name}'][@value='#{label}']")
end
end