Charlene O’Hanlon and Rob Mason from Applause talk about how crowdsourced testing is a highly effective method for testing applications and its benefits to DevOps teams. The video is below followed by a transcript of the conversation.
O’Hanlon: Hey, everybody. Welcome back to TechStrong TV. I’m O’Hanlon, and I’m here now with Mason with Applause. Rob is a return to TechStrong TV. Rob, welcome back. So good to see you again.
Mason: Thank you, and thanks for having me on again.
O’Hanlon: All right, so we had a great conversation, what was it, a couple of months ago?
Mason: Yeah, a little while ago.
O’Hanlon: I think so. Yeah, yeah, so time flies when you’re having fun, I guess. But, at that point, I did invite you to come back on so we could talk about the topic of continuous testing, which I think is something that is gaining a lot of traction in the DevOps space, and it’s certainly garnering a lot of conversation. Sorry, not continuous testing, crowdsource testing. Oh my goodness, my brain this afternoon, what am I going to do? Sorry about that. Yes, crowdsource testing, and it, too, is something that a lot of DevOps teams or a lot of organizations are considering as an alternative, I guess, or maybe a complement to the way that they’re doing testing these days. You are the expert in the field, so I want to talk to you about it and kind of get a download, if you will, on crowdsource testing. What you’re seeing in the space right now, who’s talking about it and what organizations, how they could benefit from it?
Mason: Right. Thank you. Yeah, so Applause is the largest crowdsource testing company in the world. _____ [inaudible due to audio distortion] quality testing for some of the largest companies in the world, like Google, PayPal, Walmart, Amazon, Facebook, a lot of those different types of companies, and we do that digital quality testing by using a crowd. Applause actually started off as uTest about a decade ago, _____ the name uTest, so anyone could test. Basically, if you go to uTest.com, you sign up to become a tester. Today, that community is over 750,000 people from 233 different countries around the world, and they’re making money from testing products from the company, or the companies I mentioned before, as independent contractors. They bring their own computers, they decide what they work on, what they don’t work on, they bring their gear, they decide when they want to work.
You start off by going to uTest.com, you sign up, and then we put you through all sorts of training. We have what we call uTest Academy, lots of great videos, training courses. It’s all free. You don’t have to pay anything for it. Joining the community is free. Once you join the community, you start to level up your skills as a tester, and you become better and better at writing great bug reports, with _____ the attachments, the steps to reproduce them, learn now to execute test cases, how to do payment testing or accessibility testing. We train you on all sorts of things around testing because we know there are not enough tests going on. We know there is not enough quality in our products today. It could always be better, and so ‒
O’Hanlon: Yeah ‒
Mason: Go ahead.
O’Hanlon: No, no, you go ahead.
Mason: I mean, with how fast the technology landscape is changing, you guys see it firsthand every day, it’s very easy to switch from one bank to another or from one app to another, and so it’s very important that that first experience is first class. More and more big brands are very much aware and very in tune with those first experiences and the ongoing experience of their users, so we use a very large community of real people in the real world, around the world, every country you can imagine. You know, 233 countries, as I mentioned. They level up over time to become more and more expertise. As they progress in their learning and participation, they get invited into more projects, so they get opened up to more ability to earn money, basically. When they tell us ‒ voluntarily, of course ‒ about themselves more, what devices they have, do they have an iOS device, android device, what kind of laptop they have, they get invited to more projects.
The more they tell us, the more that we can match them to projects. We use machine learning algorithms to match testers to projects based on past experience, so we’ve got a data science team, and machine learning algorithms that match based on performance. It’s not based on demographics or race or ethnicity, it’s all based on what you can do, and what you can do is behind a screen. The machines don’t know. It’s based on did you turn up on time for the project, did you execute ‒
O’Hanlon: Yeah.
Mason: ‒ _____ well, were your bugs well written? It’s all based on pure scoring, which is great. Some of our projects do require physical presence, it will be like a test for Starbucks, and you have to go in and buy a coffee and prove that you can order it online and pick it up, and all that sort of stuff. But, most of them are remote, and you can work wherever you are, from your couch with your dog sitting on your lap, and still execute great testing for the customers. That’s what the community is really a lot like. Some of the projects do have demographic requirements, but mostly those are in the medical space or in fitness. For instance, we’ll be looking for a specific age class for testing online fitness classes for some of our customers, and so sometimes there are demographic requirements. But, for the most part, it’s pretty wide open for anyone to participate. What’s nice is in some of these communities, or sorry, in some of these countries, it’s hard to find jobs. Especially during COVID, we had about 50,000 coming in per month during COVID.
O’Hanlon: Wow.
Mason: That’s trailing off now with less people coming in, but with all the people that were out of work, you can make money at Applause. Some people do it as a moonlighting job on the side, just to earn some extra money, maybe some play money. Some people make a living out of it. As an independent contractor, again, they make their own time and everything, but they’re making a living off of Applause doing testing for these customers full time. Again, anyone can do it, so if you’re in a country where maybe women can’t get jobs because it’s old-school and it’s sexist there in that country or what have you, or you’re in a demographic that it’s harder to find work, uTest doesn’t care, like bring your skills, bring your abilities, and anyone can get work. If you look at our community, we’re incredibly diverse across, again, all those countries, but also in our global community, we have a high ratio of women in the community.
Not everyone, obviously, tells us what gender they are, but 35 percent of our global community are female. We have top-rated testers in the community, 41 percent of our gold-rated testers, that’s the highest level tester we have, 41 percent of those are women, 40 percent of our silver testers are women, 40 percent of our bronze testers are women, so you can just see that across the board it’s, again, based on performance. It has nothing to do with gender or anything else, which just levels the playing field for a global economy, and for global testing of enterprise products. I’m sorry. I’ve been talking ‒
O’Hanlon: No, that’s awesome. I love to get the download, so it’s like you kind of take it from there and just run with it. Wow. That’s really cool stuff. I think that the last time we spoke, you guys were working on a couple really cool projects, but it seems to me that this would be a great technology, or a great service, if you will, for organizations that really don’t have maybe an in-house testing environment, or maybe organizations that do have an in-house testing environment but really kind of want to go that extra mile and make sure really that all their bases are being covered. How much of your customer base, then, are those companies that maybe already have testing abilities in house but they’re just looking to kind of up their game, if you know what I mean?
Mason: I would say probably almost all of them have QA ‒
O’Hanlon: Really?
Mason: ‒ of some sort, but the difference is QA in a lab with a relatively small team is very different than real world QA. If you have some simulators or some real devices in the lab, some browsers based in your lab, it’s very different than browsers around the world. Probably a lot of the enterprise companies have their sites localized into different languages. Probably in your lab in your local team, you may not have all those native language speakers, so that’s _____, so localization is one. Currency testing, payment instrument testing, all those things that happen in the real world, you know, poor networks, flaky networks, different flavors of Samsung devices or android pixels, all that sort of stuff. A lab test is great, and it’s a great way to make sure the base foundations of your applications are good on your mobile apps, but it’s very different when you get out in the real world, and that’s where the customers judge you.
They don’t judge you for your lab experiences, they judge you when you get out in the real world, that’s important. Almost all of our enterprise customers have their own QA teams, but they admit, and these are big companies ‒ it’s Google, it’s Amazon, it’s Facebook ‒ they admit that they don’t have a big enough team, even at their size, to cover what they really need to do to test their applications. You need a company that can bring resources from around the world, speaking all the different languages, all the different currencies, all the different types of devices. I think the last time we counted, there were about four million different devices in the hands of this community, because everyone ‒
O’Hanlon: Wow.
Mason: ‒ has got a phone _____ [inaudible due to audio distortion], what have you, so you take all those devices and you bring them to bear on some of these huge problems of the large enterprise, and it changes the dynamics. It’s daunting if you want to build a product and launch it globally and have it be successful, and not know that it’s going to work the first time. Even if you get a great, large outsourcing firm based in India or over in Europe, they’re based in one area, in one region. They all speak probably the same language, a very similar language, they’re in the same area, the same currency. How do you get that global test? The answer is crowdsource testing. There is no other way to do it. It’s just not practical any other way. If you have to manage those masses of people, like how do you make sure that it’s not just total chaos, and that’s what Applause has really specialized in for more than a decade, is making sure that we give great quality in what we give back to our customers, it’s actionable, it’s precise. Every bug report from us has to have an attachment and a video, it has to have reproduction steps.
Again, we don’t just do bugs for customers, we do all sorts of testing, like we test smart phones, we test smart TVs, smart cars, IoT, AR/VR. We do machine language, data collection, data collection from machine learning ‒ I said machine language, but machine learning ‒ so all sorts of things we do for our customers using this global community. When you think about it, I have 750,000 people I can bring to bear on your problems to improve the digital quality of your applications. Sometimes you need more data, sometimes you need _____ [inaudible due to audio distortion], sometimes you just need a lot of people to bang on a website, sometimes you need people to go deep and look for those hard-to-find bugs. It varies greatly, so we cover pretty much every area that you could expect of digital quality testing with that crowd, and we _____ [inaudible due to audio distortion] so you don’t have to think about it. That just allows us to scale, but what we’re really proud of is just how successful our community has been, how diverse they are, how broad and how deep they get.
They are true experts, and a lot of these people in the community, some of them are just people moonlighting after working some other kind of job, but some are true QA experts that have been in the industry for decades and have just chosen this new way of life, because it’s flexible. You always work at home, or wherever _____ [inaudible due to audio distortion]. You can be on the beach and doing the job. It doesn’t matter to us, as long as the job gets done. It is the new world, and as COVID removed some jobs for some people, at least temporarily, we got an influx, but also people are now looking for that new way of working. We see the industry moving out of the offices, and that’s what we’ve been at Applause for a decade. Our community, the people doing the testing for us, have been outside of offices since we started, and so it is the natural way for us. It’s like welcome to our party, but now you have to figure out how to work ‒
O’Hanlon: Right.
Mason: ‒ _____ [inaudible due to audio distortion], and we’ve been doing that _____ for a long time.
O’Hanlon: That’s very cool. On average, how many testers do you have on a particular project?
Mason: It varies greatly, like sometimes we’re doing a global payment instrument test that’s making sure something like an android pay or an Apple pay works across the globe when they’re rolling out a new feature. There, we have to engage a lot of people to do that, to make sure that all the different banks, all the different currencies work around the world. Sometimes, it’s a point release, a patch release from a customer with a new version of their mobile app, they made some targeted changes, they’ve told us what they are, some regression testing around the area, a specific testing in that area, so it’s a smaller team, so anywhere from a handful of people to thousands of people at times. We don’t generally go much larger than thousands on a single project unless there is a very specific need. We do have some that spike up there in very large numbers, but, for the most part, we’re in the tens to hundreds range, less so in the thousands on a given, specific, what we call test cycle.
For most of our customers, in fact all of our customers, it is subscription-based at this point, so they’ll sign up and have us test a digital asset over a long period of time, and so those assets go through their cycles. The first release is different from a patch release versus a new, major update, a new, major feature. Again, the size of those teams vary over time, and we can pull as many as we need into it. If there are different skill sets, like maybe a customer has a mobile app and now they’re focused on accessibility, and so they’re going to do a release 100 percent focused on making their application more accessible, we’ll bring in an accessibility group from our community and have them focus on accessibility. These are people not only that know accessibility testing well, but they’re people with real accessibility limitations, so it’s real people with real disabilities testing your application to make sure it works in the real world, because there is nothing more frustrating than trying to use an application that claims it supports you and then finding out it doesn’t.
O’Hanlon: Right, right. Well, it sounds like great stuff, and you guys have been doing this for ten years or so?
Mason: Yeah, just over 11, coming on 12 now.
O’Hanlon: That’s great. How did you guys come up with this idea? I mean, I think it’s really kind of fascinating because it really does address a real problem that is in the industry, and that is being able to test all the things, if you know what I mean. Is this something that your founders just decided one day like, “Well, you know what? The best way to do it is let’s just ask everybody to do it, and everybody to help us out.”
Mason: Yeah, our founders were engineering managers from the past, they were engineers, and they had the experience of building and testing applications in their own labs, and then going out in the real world and finding out that things were different when they got out to the real world. They said, “We have to change the model. The model isn’t going to work, and it’s only getting worse over time.” They said, “We have to go and turn this model upside down,” and they were far ahead of their time when they launched the company. Now there are some other much smaller companies out there doing some similar things, but they’re much more niche. But, they saw what was coming. The real world matters, and the differences are just ballooning and you just cannot keep up, even as a large company in the lab, so great foresight there. It took a lot of work to get to the point where you can manage that many people and have it be successful.
Obviously, if you just put random people on a project and say, “We’ll do stuff for you,” you’re going to get terrible results and just waste your time. Obviously, places like Walmart and Facebook and others like that will not tolerate that, so you have to have a very good system and process around it. We have a special enterprise grade platform that our customers and community interface on, so we do software development to make our own platforms to allow those groups to interact with each other, and then we’ve got a great team of community management around them, recruiting people, training them, helping them learn how to be great testers. Ultimately, we’re making careers for people. We’re helping people make livings working for themselves wherever they want to be.
O’Hanlon: That’s great, and we’re going to need to wrap this up here very soon, but have you heard of any stories of folks that joined the platform to do testing here and there, and ultimately decided that, “Hey, this is what I want to do for a living. This is what I want to do all the time,” and they ended up actually doing a career change because of this?
Mason: Absolutely. In fact, we’ve hired several of those.
O’Hanlon: Oh, really?
Mason: It’s not generally the model, to hire from the community, because the community is our model, but sometimes they rise up so far and they’re such stars in the community, and we’re looking for additional help in managing the community, or elsewhere in our business, that we find them and bring them in. A lot of our great employees we have at the company are people that we found through the community, because they’ve come in and shown what they can do. There are others that have gone off and moved on to full-time positions in companies. Some of them do have full-time QA positions, and then work for us moonlighting. Some of them, like my daughter is in the community, anyone can be in the community. My daughter is in the community, and she makes a few bucks on the side and buys herself snacks or whatever _____.
So, it really is a very diverse community of everyone who is just getting started or learning about testing, giving an appreciation for what it takes to make digital quality products, and what companies go through to test them, so I think it’s a great experience for everyone. I encourage every single person to sign up at uTest.com. Again, it’s free. We pay you, not the opposite, so you can make money. All you have to do is experience some digital quality products, give us good feedback, and help us make these products better for the planet, basically.
O’Hanlon: That’s awesome. Well, Rob, thank you so much for walking me through this. I love the idea of pretty much crowdsourcing anything, because I truly believe in the collective intelligence of the populations. Anything that we can do to kind of encourage that and really take advantage of that at the end of the day, really capitalize on that collective intelligence and the ability of crowds to really do some good, I think is good for everybody. Thank you, again, and it was great having you on TechStrong TV again, and hopefully we’ll see you again in the future.
Mason: My pleasure. It was great talking to you.
O’Hanlon: All right, everybody, please stick around. We’ve got lots more TechStrong TV coming up, so stay tuned.
[End of Audio]
