A Quick Evaluation of Testing Frameworks : Crosscheck

Been checking out the Crosscheck Framework. It is a open-source testing framework for testing your javascripts. It is a quite ordinary unit testing framework but the thing is that it, unlike frameworks as jsunit, runs without a real browser. Instead it simulates some of the best known browsers such as Firefox and IE. According to what I heard it actually do it quite well also. So what is the major benifit of this, you may ask. The answer is speed. It allows you to run cross-browser check in a couple of seconds. That also means that we can integrate our javascript tests with our Continuous Integration system Cruise Control.rb. A major benifit. Especially, as in this cae, when it could be done on multiple browser engines.

The main drawback (for us) with this framework seems to be difficulties about how we should integrate it with the rest of the testing system. Crosscheck has some support for loading the DOM and then use that for tests. The problem is that we need access to rest of the testing enviroment to generate the right DOMs. There might be some obvious solution to this that I just missed. I have to think about it. Crosscheck is written i java, so for a while I was thinking we should run it from JRuby. This makes it possible to both access all of crosscheck and all of our rails system. The problem here is that it may introduce to many bugs.

Alternative some simple patch could solve the problem for us. I have to return to this subject later. Any ways, crosscheck seems to be a real great alternative for us and I'm definitely going to trying out. Soon.

A quick evaluation of testing frameworks: Rbehave

My company is developing it's product using the web framework Ruby on Rails, developed by 37signals. This framework contains good support for regular unit testing but SoundCloud decided to use the behavior-driven framework Rspec for their testing. (For more on behavior-driven-testing, bdd, read by blog post about it here).

This framework originally has no support for integration testing but in the next release they will integrate rbehave which is a nice framework for writing "user story"-like tests. (User story beeing the agile version of use cases, which are formulated as a short story). Here comes an example of rbehave from another blog

Story "View Home Page", %{
As a user
I want to view my home page
So that I can get a birds eye view of the system
}, :type => RailsStory do

Scenario "Publisher with no videos" do
Given "a company named", "No Videos" do |name|
@company = Company.create! :name => name
end

And "a user named", "novideos" do |login|
@user = create_user login
end

And "the user belongs to", "company", "No Videos" do |klass, company_name|
@user.update_attribute :company, klass.classify.constantize.find_by_name(company_name)
end

And "logged in as", "novideos" do |login|
post "/sessions", :login => login, :password => "test"
end

When "visiting", "/" do |page|
get_via_redirect page
end

Then "viewer should see", "companies/show_initial" do |template|
response.should render_template(template)
end
end

The rbehave framework, which essentially is a port of the user-story based java testing framework Jbehave, is very fresh and most likely still has some problems. It seems for example, by reading a couple of blog posts (such as this) , that there could be a bit of a problem to get the framework to work with Continues Integration (CI) which soundcloud uses.

Otherwise this seems like an ideal framework to use, and as soon as the next release of Rspec we will start using it I think. (Maybe even sooner )

The Thesis and It's Goals

The scope of this thesis is to secure the company's website. Securing in this context has nothing (or very little) to do with security. In this sense it means making sure that the product satisfies it's specifications, i.e. keeps the promises me & the rest of the SoundCloud crew gives :) .

So lets take a look at how the current state is in the company. One of the main goals to achieve a stable product is to integrate testing in the development process. Following the agile methodology, the company uses so called test-driven-development (tdd), which I think is a great way to make the product more stable. (For more on test-driven-development, read my blog post about that here). The current state is that the company has quite a good test suite on the backend side but absolutely none on the frontend side. It also lacks some sort of backend integration tests. By this I mean that there exists tests that test certain modules and their functionality but there lacks tests that test the integration of these modules, i.e. that they function togheter.

My hopothesis for this thesis project is therefore that the company needs to adopt three additional testing strategies. First : the backendside needs integration tests for almost all of it's modules. Second : the frontend side, i.e. the javascripts, needs some kind of unit testing suits to verify that the javascript works on a function/method level. Third : the product needs some kind of acceptence test framework in place, i.e. a tool that verifies that the final product is working as it should.

The main goal of the thesis it to find a way to integrate these testing strategies in the development process along-side with the current testing framework. One issue that arises is that the findings must be easy to adopt and that the tests and their frameworks must be easy to maintain. By easy I mean that the tests must give more stability and reability to the procuct than they produce overhead by the need to correct tests and keep them up to date. The ideal solution would be to find ways to integrate the new needed tests in the companies current testsuits and workflow.