Crowdsourced mobile application testing: interview with Mob4Hire

When developing for mobile phones there are a number of different variables which can make effective testing time-consuming and difficult. The end-users of mobile applications and services will be using different networks, phone models and potentially different software versions which can all influence the user experience.

Mob4Hire is solving this by enabling developers to post projects online with an accompanying testing specification which an open community of testers around the world can then bid for. This gives developers potentially unlimited access to different test configurations while minimizing the cost involved.

Sony Ericsson Developer World met Paul Poutanen, President and founder of Mob4Hire, who answered our many questions. Below is this conversation:

Where did the idea come from and what is the story behind Mob4Hire?
The idea for Mob4Hire began in 2005 when I was launching the first location-based game in North America. While the software development was going fine, I could not believe the hassle and the expense of testing the application on various networks and handsets. I had to fly to the US with a small team of developers and purchase handsets so they could test their application on a US network.

Initially we couldn't even buy the handsets and subscribe to the mobile service because we didn’t have a US address. Eventually the tests were completed successfully, but I realized testing was the most difficult and onerous component in the whole development process.


Paul Poutanen, President and founder of Mob4Hire.

We were all exhausted, sitting in the airport waiting to board a flight home, when I saw a guy chatting on his cell phone. He was using a handset identical to one we had just purchased and tested on. It dawned on me that if I knew him I could have just sent him our application and have him quickly run the test for us. I'd be happy to pay him for his trouble and it would save me a load of time and money.

That was the light bulb moment. That's when Mob4hire was born. I had experience with a crowd-sourcing company in Calgary that had a mandate to nurture business ideas using the power of the Internet "crowd". That's exactly what Mob4Hire does.


Explain how the concept works. What's your business model?
The concept is based on the idea that there are many able, technical people around the globe who would gladly test mobile applications for a fee. They already have the subscribed handset, the time and the inclination. These are typically people who can be defined as the early-adopters of the mobile application industry. Mob4Hire connects those people with developers who want to test their applications on carrier networks where the tester lives.

Geography is the key. Testers need to be physically located where the carrier's service operates.

Testers bid on the testing projects posted by the developers at Mob4Hire. The developer selects the testers they want, downloads their application to the testers' handsets along with specific testing criteria, and pays for the tester's service once they receive the feedback they've requested.

We launched our Beta site in April of 2008. Currently Mob4Hire has close to 1000 handsets ready to test applications in 37 countries on 87 different networks. This includes virtually every handset and network in North America.

The ultimate result is we are able to help speed up the application development cycle in a very cost-effective manner.



Who are these testers, and how do you know they will provide good service?
Testers are generally early adopters who can appreciate the value of a new application and know a bug when they see one. It's the developers responsibility to clearly outline the steps they want the tester to follow. While there is a certain amount of trust that is part of the system, most developers can tell if the application has been tested by the quality of the test report.

All funds are held in escrow until five days after the completion of the test so any tester that has not done the required work will not be paid.

Mob4Hire uses a five-star rating that rates both the developer and the tester after each completed test. This will allow the best testers to shine and quickly build a reputation in the developer community.

We also discovered that many of the handsets were registered by professional testers. It turns out many developers were interested in turning their expensive testing departments into a profit center, bidding on outside jobs to keep their idle testers busy. We've responded to that by developing our Mob4Hire Certified Tester program which is an extra level of comfort for developers looking for the best possible testers.


What are the costs involved for using the service?
That's determined by the market. Testers bid on projects and developers decide which bids to accept. Because data rates vary dramatically from country to country, Mob4Hire is using a bidding process similar to the eBay business model.

We are convinced the bidding process will ensure that the application developer gets a fair price because they can easily see range of all the bids. If there are several Mob4Hire testers bidding for the same job - that gives a strong indication of the market value for that specific test.

Mob4Hire collects a commission from each transaction. This ranges from 6–10% of the cost of the test depending on the number of handsets that the developer has placed in our system.


The internet already has communities of people prepared to test applications and provide feedback at no cost.
While there are many free download sites on the internet, we firmly believe that fees for services rendered is the foundation of an effective free-enterprise system. Our experience is that most free download sites will get your application used by a number of people, but very seldom are they motivated enough to provide the developer with useful information or even bother to report the bugs they have found.

There is enormous goodwill out there, but not many companies are prepared to base their business decisions and development cycles on enthusiasts who may or may not offer valuable feedback.


Crowd sourcing is a relatively new term. How do you define it, and how are you tapping into the crowd?
Crowd sourcing takes a task traditionally performed by an employee or contractor, and outsources it to an undefined, generally large group of people, in the form of an open call.

We believe that some crowd-sourced ideas fail if there is an underlying assumption that individuals will perform quality work on time without immediate and fair compensation.

Mob4Hire was formed with the concept that we could gather skills from around the world and build an effective commercial entity.

Let's face it. We are all busy people and if something has to give, it is usually the item on the plate that we have the least obligation to perform. Putting a commercial face on this concept spins it around and makes it work.


There are many companies who have established in-house testing teams. How can crowd-sourced testing work with an existing testing process?
This is a great opportunity for both Mob4Hire and the developers who have an established in-house testing team. We really provide an opportunity to turn their current testing department (which is most likely a cost-center) into a profit center. We have found that it can really fit into their current business plan because it allows them to test for others when it is convenient, while still maintaining an effective testing department without the accompanying financial burden.

Currently approximately 50% of Mob4Hire's handsets are in the hands of professional testers.


What type of testing services are available?
A test could be as simple as a ping test to see if the application is blocked by a carrier in a foreign country, or it could be an extremely complex test that may take 100 hours to complete. The bidding process determines the worth of these projects.

We are also developing a separate system whereby the developer can remotely log on to the phone and conduct the test themselves. All that would require is that the appropriate handset be turned on and operating on a carrier's network. This doesn't erase the need for a geography-dependent crowd of testers, but will put the testing capability into the developer’s hands.


If I submit an application for testing, how can I ensure that the tester will extensively test the application according to the test specification?
Developer's will benefit by providing a carefully drafted test plan. This serves two purposes. It will attract the appropriate testers and enable them to bid for the project based a clear understanding of the work involved. It also establishes the developer's expectations and clearly outlines the steps the tester needs to follow in order to fulfill the agreement.

Like any business relationship, an element of trust is required. Mob4Hire's online rating system allows developers and testers to offer feedback on the quality of their interaction. It serves both parties to meet each other’s expectations. A highly rated developer will receive more bids, and a highly rated tester can expect to receive more projects.


If there is a dispute over the testing, how is this mediated?
If there is a dispute over the testing, Mob4Hire will continue to hold the funds in escrow and then Mob4Hire personnel will intervene and mediate. Our goal is to ensure that the quality of testing remains very high.


What are steps required to go through a complete test cycle?
The first thing for a developer to do is register with Mob4Hire. When their project is ready to be tested, they post it on the Mob4Hire "site" with a complete test plan, specifying which handsets and networks they want tested.

Bids are entered by the testing community and are hopefully accepted by the developer, although there is no obligation to do so if the bids are unsatisfactory.

The successful bidder receives an SMS with a URL attached that gives him/her access to the application. The application is downloaded wirelessly and the test plan is completed. The developer then has five days to review the completed work and decide whether the test report was completed to his/her satisfaction. If we do not hear from the developer, we assume that all was fine and we release the funds to the tester.


How do you see the relation of Mob4Hire testing to other testing programs, such as Java Verified and Symbian Signed?
Java Verified and Symbian Signed tests may work very well on specific handsets on specific carriers but not on others.

For example even though a developer has been verified or signed does not mean that a network connection will work on a specific handset. A carrier/network operator may block network access to a specific URL that is directed to a mobile server.

As well, just because an application is verified or signed does mean it is a good commercial application. The GUI may be not be intuitive or the application too complicated. The end user may find the help text does not provide enough help or lack detail in given areas.

Mob4hire can be used before verification or signing to test out an alpha version for marketing reasons. As well, it can be used after porting to multiple platforms and handsets to test for functionality.


How can this type of approach be used for marketing reasons?
There are a number of issues that will affect the sales of an application. Do people know what your application does? Is it intuitive? Does the GUI make it difficult to use? Is the help screen truly helpful? Would end users use purchase this application? At what price?

Mob4hire can be used as a focus group for a select group of testers who can answer a series of questions on the applications. The testers would not necessarily be looking to test for functional testing but how they liked the application. The testers may make recommendations about the application that would benefit the developer.


What the technology trends see getting the most benefit from this type of crowed sourced testing?
Mob4Hire is network, carrier, handset and platform agnostic and provides a service required by all developers, regardless of their applications and technologies. Essentially, we are acting as a broker between the testing community and the mobile application developer community. Therefore we think that any mobile application that requires testing is a potential customer for us.

That being said, we think that the LBS market is of special interest because of its inherent need for true mobility testing and its potential upside for growth. We also hear a lot from the social networking application companies as they want to expand their networks across networks and geographical areas.

Ultimately, the nature of the application is less important to us than the quality of the testing that is conducted through Mob4Hire.


How do you see the service developing in the future?
Mob4Hire's goal is to become the central meeting place for the application developers and the mobile testing communities. These two communities require ongoing dialogue and we exist to facilitate that relationship. We are excited to see what develops from this improved interaction and intend to provide additional services to further improve those relationships.

The industry is witnessing the convergence of mobile voice and mobile Internet. The applications that will come out of this ground-breaking environment are only limited by the imagination of the developers. Mob4Hire exists to accelerate that development process.