Meeting with the Ableton CTO.

Our team had the opportunity to meet the CTO of Ableton and get some general advices. He was of course a real clever guy who knows a great deal about developing software. We got to discuss testing and quality with him and they have it all: testdepartment, testframeworks, inhouse developed acceptence tests, etc. They also had a small group of dedicated users that received prereleases to evalutate. This proved to be a valuable things, not only for discovering bugs and see what needs to be changed, but also because they tend to hype the next version of Abletons software.

Something intresting that came up was that before they had a separate testdepartment they had this one guy who always managed to find the most bugs in their software. He applied the infamous 'monkey test strategy', i.e. randomly klick on everything as much as possible. Surprisingly this, as I said, proved to be the most successful way of finding bugs and he thought it could be a good idea to program some kind of 'bot' who randomly presses everything. But if I should do that I have to find a way of recording what the bot done, otherwise it would just come up with a bug and left us with no clue how to reproduce it.

Lets stop and think about this. This would be really intresting to try out. The way I imaging doing this is to have a couple of conditions that always has to be satisfied and then let the bot randomly visit the site and check that these conditions are met. One example of such a conditions is the accessability of tracks. We have provide the ablitiy to upload and share tracks on our site and the user can choose which persons to share their tracks with. This is of course quite errorprone and very bad for our reputation if we fail to keep the private tracks private.

The next problem is to record what has happend, Ableton has solved this by secretly recording every event in their program. If the programs then crashes they could just replay the events to reproduce the bugs and see what went wrong. Nowadays this is very helpful because they can let the users send them their crashing program and just replay it to find the bug.

Another part of quality, that we discussed, is planing. According to the Ableton CTO one sure way of introducing bugs is to plan to much features to the release. His general idea about this was that it's better to have very few really good features that works very well than to have very many features that work ok. This could seem obvious, but in reality it's not. There are so many things you like to cram in there and everybody knows that it hurts to 'kill your darlings'. Then comes the thing that people tend to underestimate how long time things take to develop. According to the CTO only 10% of software projects are finished in time, and I think none are finished with all feautures that were planed for it.

Oh and another tip we got was: merge often, release often.

A quick evaluation of testing frameworks : htmlunit

I had a quick look at htmlunit. It's a kind of a browser emulator, it parses html sites for you and even has the abliity to run some javascript. Then you could 'click links', 'fill in fields' and 'press buttons' on your website by calling some functions on the emulator.

The framework, which is written i java, seems kind of "oldschool" and doesn't seem to handle all the ajax we're doing on our site.It has some support for (XMLHTTPRequest) but that is kind of it. One nice thing about it is that it doesn't provide testingmethods (such as assertions) so you have to use an external testingframework to do this. This lets you use the same framework for all of you application and homogenise your tests. Nice.

Because we're using ruby on rails to develop our site and htmlunit is written in java we could have some trouble integrating it and probably have to use JRuby to get it to function. This is doable but think about it, first were using JRuby (a ruby runtime implemented in java) instead of ruby, this could be the cause of numerous bugs. Then on top of this we should use a browser emulator instead of a real browser, which will introduce even more bugs and glitches that, in reality, isn't there. The problem with this framework for us seems to be that we could spend to much time 'chasing ghosts'; even if the tests fails that doesn't mean that we have a bug. This is of course always a problem with testing, but this seems even worse.

Instead of this framework I stumbled over the Hpricot_forms framework. Hpricot is a nice framework for parsing html/xml content and the hpricot_forms is kind of a htmlunit port for ruby. It's really nothing fancy but it lets you "click links" and "fill in and submit forms" in your tests. I downloaded the project and after some troubles, including manually updating some code, I had it running and wrote some tests in it! It's nice and really supply a valuable addition to our testframework Rspec, but I think it would be even better if combined with the Storyrunner included in the next Rspec release.

A Quick Evaluation of Testing Frameworks : Selenium

So it seems that everybody that's coding rails and ajax stuff is talking about Selenium. So what is all the fuzz about?. Selenium is a acceptence testing framework, i.e. a tools that you use to test the final product.

The idea is to tests things like : if I click on the inbox button my messages should show. When I'm on the inbox page I should be able to click on the 'mark as read' - button and the message should be marked as read.

To people doin' agile development this may seem familiar; it resembles 'user stories'.

The tests seems easy to write and Selenium seems to be easy to integrate with rails, there is even a selenium-on-rails project. I've heard some people complain about it, saying that it is to much work and that the tests run to slow. I just had a quick look at it but to me it seems to be a good alternative. I mean, this tool runs in almost any browser and clicks the buttons for you. What is the alternative? To manually click on every single button/link/thing in every supported browser. That seems very tedious...

Any ways, the benifit of the this tool is the what-you-see-is-what-you-get feature. As with any browser-boot, it actually hijacks your browser and surf the site for you. It the site works for selenium it works for you. The major drawback seems to be speed. I wrote 3 simple tests and then ran them 200 times, equals 600 tests. This took me 11 minutes! Compare this to the 500 backend test we have, which run in about 30 seconds... We could easily have around 1500 tests in a year and then it takes 30 minutes to run them. This really is a drawback. One other thing is that it requires a GUI to run, which makes it hard to integrate with the continuous integration system.

After discussing the issue in our company it seems that the benifits seems to be greater than the drawbacks. We just bought a dedicated testing machine and hooked it up with a couple of operating systems and different browsers. The idea would be to have selenium running there now and then. And when it comes to speed : what are the alternatives? We have to test the application.

Considering the 'user story' nature of selnium it would seem like an ideal idea to integrate it with the Rbehave framework I described before. And after surfing the web for a while it turns out that this has actually been done. It still seems a bit early and 'cutting-edge' but the result is nice (taken from Kerry Buckley):

Story: Login
The front page should contain a login form

Scenario: Loading the home page when not logged in

When the user goes to /
Then the title should be 'mojo'
And the page should contain the text 'Welcome to mojo'
And the page should contain the text 'Sign in using your OpenID'
And there should be a field named 'openid_url'
And there should be a submit button named 'login', with the label 'Sign In'

steps_for(:selenium) do
When "the user goes to $path" do |path|
$selenium.open path
end
When "the user types '$text' into the $field field" do |text, field|
$selenium.type field, text
end
When "the user clicks the $button button" do |button|
$selenium.click button
$selenium.wait_for_page_to_load 5000
end
Then "the title should be '$title'" do |title|
$selenium.title.should == title
end
Then "the page should contain the text '$text'" do |text|
$selenium.should have_text_present(text)
end
Then "there should be a field named '$field'" do |field|
$selenium.should have_element_present(field)
end
Then "there should be a submit button named '$name', with the label '$label'" do |name, label|
$selenium.should have_element_present("//input[@type='submit'][@name='#{name}'][@value='#{label}']")
end
end




A Quick Evaluation of Testing Frameworks : Crosscheck

Been checking out the Crosscheck Framework. It is a open-source testing framework for testing your javascripts. It is a quite ordinary unit testing framework but the thing is that it, unlike frameworks as jsunit, runs without a real browser. Instead it simulates some of the best known browsers such as Firefox and IE. According to what I heard it actually do it quite well also. So what is the major benifit of this, you may ask. The answer is speed. It allows you to run cross-browser check in a couple of seconds. That also means that we can integrate our javascript tests with our Continuous Integration system Cruise Control.rb. A major benifit. Especially, as in this cae, when it could be done on multiple browser engines.

The main drawback (for us) with this framework seems to be difficulties about how we should integrate it with the rest of the testing system. Crosscheck has some support for loading the DOM and then use that for tests. The problem is that we need access to rest of the testing enviroment to generate the right DOMs. There might be some obvious solution to this that I just missed. I have to think about it. Crosscheck is written i java, so for a while I was thinking we should run it from JRuby. This makes it possible to both access all of crosscheck and all of our rails system. The problem here is that it may introduce to many bugs.

Alternative some simple patch could solve the problem for us. I have to return to this subject later. Any ways, crosscheck seems to be a real great alternative for us and I'm definitely going to trying out. Soon.

A quick evaluation of testing frameworks: Rbehave

My company is developing it's product using the web framework Ruby on Rails, developed by 37signals. This framework contains good support for regular unit testing but SoundCloud decided to use the behavior-driven framework Rspec for their testing. (For more on behavior-driven-testing, bdd, read by blog post about it here).

This framework originally has no support for integration testing but in the next release they will integrate rbehave which is a nice framework for writing "user story"-like tests. (User story beeing the agile version of use cases, which are formulated as a short story). Here comes an example of rbehave from another blog

Story "View Home Page", %{
As a user
I want to view my home page
So that I can get a birds eye view of the system
}, :type => RailsStory do

Scenario "Publisher with no videos" do
Given "a company named", "No Videos" do |name|
@company = Company.create! :name => name
end

And "a user named", "novideos" do |login|
@user = create_user login
end

And "the user belongs to", "company", "No Videos" do |klass, company_name|
@user.update_attribute :company, klass.classify.constantize.find_by_name(company_name)
end

And "logged in as", "novideos" do |login|
post "/sessions", :login => login, :password => "test"
end

When "visiting", "/" do |page|
get_via_redirect page
end

Then "viewer should see", "companies/show_initial" do |template|
response.should render_template(template)
end
end

The rbehave framework, which essentially is a port of the user-story based java testing framework Jbehave, is very fresh and most likely still has some problems. It seems for example, by reading a couple of blog posts (such as this) , that there could be a bit of a problem to get the framework to work with Continues Integration (CI) which soundcloud uses.

Otherwise this seems like an ideal framework to use, and as soon as the next release of Rspec we will start using it I think. (Maybe even sooner )

The Thesis and It's Goals

The scope of this thesis is to secure the company's website. Securing in this context has nothing (or very little) to do with security. In this sense it means making sure that the product satisfies it's specifications, i.e. keeps the promises me & the rest of the SoundCloud crew gives :) .

So lets take a look at how the current state is in the company. One of the main goals to achieve a stable product is to integrate testing in the development process. Following the agile methodology, the company uses so called test-driven-development (tdd), which I think is a great way to make the product more stable. (For more on test-driven-development, read my blog post about that here). The current state is that the company has quite a good test suite on the backend side but absolutely none on the frontend side. It also lacks some sort of backend integration tests. By this I mean that there exists tests that test certain modules and their functionality but there lacks tests that test the integration of these modules, i.e. that they function togheter.

My hopothesis for this thesis project is therefore that the company needs to adopt three additional testing strategies. First : the backendside needs integration tests for almost all of it's modules. Second : the frontend side, i.e. the javascripts, needs some kind of unit testing suits to verify that the javascript works on a function/method level. Third : the product needs some kind of acceptence test framework in place, i.e. a tool that verifies that the final product is working as it should.

The main goal of the thesis it to find a way to integrate these testing strategies in the development process along-side with the current testing framework. One issue that arises is that the findings must be easy to adopt and that the tests and their frameworks must be easy to maintain. By easy I mean that the tests must give more stability and reability to the procuct than they produce overhead by the need to correct tests and keep them up to date. The ideal solution would be to find ways to integrate the new needed tests in the companies current testsuits and workflow.