The Art of Design Validation
These are more or less my private thoughts. If you want to check out my business projects, head on over to twentypeople.com, pareer.com and fruji.com. Questions? Reach out via Twitter, ping me @mittermayr.
It is a classic trap for almost any recently-incorporated, hyper-passionate startup: The angst around not entering the market fast enough, the urge to go wild and code like a monkey on speed, to just do whatever it takes to push out the first release. People need to see the product, and once it’s live, they will totally love it, right? Right!? — Well, not so fast, intrepid entrepreneur! Here is why you might be missing a highly powerful but little talked about methodology: The Art of Design Validation.
You might say that you’re different, you see the product, you envision it and most importantly, you absolutely know your customers. Of course. After all, why else would you do this? Truth be told though, there’s a pretty high chance you’re just hoping to get lucky. Even usability maniacs like 37signals have recently embraced the power of testing design concepts. Why? Because they make a lot more money with it!
Look at it this way: If you’re a guy, there will be a point in time when you get lost, with your girlfriend right next to you in the car, suggesting that you pull over and finally ask someone for directions (Why Don’t Men Ask Directions?). While (as a horde of guys is threatening me to better write that) it is clearly an insult to the power vested in us, the male population, made to hunt for food and totally know directions — our reluctance to give in, frankly, is just a clear statement that we actually got totally lost and don’t want to admit we could probably use some help.
And that’s exactly what is happening with our self-perception of being a perfectly good designer and developer, and probably among the most creative people this era has seen. Fact is: most of us are not that brilliant. It’s motivating to think you’re the Bear Grylls of Design, but the odds are against you (especially if you are featured here, here or here). And if you don’t happen to have Jony Ive doing some design freelancing for you, here is what you should be doing instead:
One of the great luxuries large companies frequently take advantage of is an array of research methodologies typically not available for the smaller budget. Focus groups, surveys, demographic screeners, conjoint analysis, EMEA, cultural perception, customer segmentation studies — you name it. They all do it, albeit only some companies, like Microsoft, openly admit it. Even the Huffington Post did it (A/B testing their headlines). Most companies though, aren’t very open about these tactics (as Steve Jobs once said, “We do no market research”). Yeah, right.
Research that fits your budget: With almost no money at hand, you can still get some serious, highly professional customer research done. There is a whole world of free (or really low-cost) prototype, design and idea concept testing tools out there. Most of them are great implementations of crowd-sourcing projects. Some of the tools are specifically made for concept testing, like PickFu, AllOurIdeas, FiveSecondTest, Concept Feedback, usaura, Feedback Roulette, Qriously and even Google offers free help. These tools provide incredible feedback in less than a day’s wait. And others, like UserTesting.com, Feedback Army and Amazon’s Mechanical Turk, might cost you a little bit, but can deliver, with amazingly fast turn-around and large datasets of feedback.
The Experiment
As an experiment, I decided to spend $5 and about an hour of work. I programmed a simple survey question and posted it on Mechanical Turk. And I got back plenty — over 100 responses plus some 30 comments of individual feedback. Not surprised by the numbers? Well, try finding a hundred individuals elsewhere, in an hour, in front of their computers, expensing their valuable time at $0.05 a pop for the effort.
So, what do you test? You can do a lot more than just simple A/B testing or short surveys. Start a crowd-sourced brainstorm, do your own affinity exercise or simply have people visit your site and respond with a simple comment on what they didn’t like about it. Paraphrasing from the 37signals experiment mentioned earlier, Jason Fried’s mantra while testing was: “We need to test radically different things. We don’t know what works. Destroy all assumptions. We need to find what works and keep iterating.” And that’s exactly what you should be doing as well. Go and experiment.
There is a whole set of arguments one could have over this. I’ll address two of them here:
- How representative is a random sample on Mechanical Turk?
Answer: It’s going to be early adopter types, not the most representative for a large operation — but we’re talking budget here. Make sure you add an optional test question; increase the number of responses you’re shooting for and maybe cut those who didn’t answer the optional question. - Why would I need this in the early days? This sounds more like a post-prototype methodology to me.
Answer: You need this right from the beginning and even as early as when you’re designing your logo. Create two versions, put them up and run it by a hundred people. Have them vote on the one they associate with your name, and let them explain what comes to their mind when they see it. Your logo, after all, will stick with you for at least a few years, if not longer. You will be very surprised.
Keep in mind: When doing (budget) research, the rule is to always remember that your participants are unlikely to have strong opinions about things that have happened in the past, and are also unable to predict the future. Your focus should be on today and on one situation, which you’re putting right in front of them for their reaction.
Are any startups using this methodology already?
You bet. I’ve asked a lot of startups to comment on this Quora question and received some great responses: “We crowd-source every change we make to our site. A/B testing our website is of critical importance to the success of our company. We need to know which page designs get customers through the door — and which don’t”, says Sander Daniels, from thumbtack.com. “People love answering the questions and we get valuable real time feedback that plays in our iteration cycle”, says Benjamin Young, CEO of nexercise.com. Or, a comment from Alex Schiff over at Fetchnotes.com, “We learned a lot about what resonates with users and that brought us to our current iteration.”
Unfortunately: If your idea doesn’t fit the market, design validation will not make that overnight success happen for you either.
