Intent to Solve

I wrote an article about finding out whether prospective customers will buy your product over at Boxes and Arrows. You should read it!

"When we’re building products for people, designers often do something called “needs finding” which translates roughly into “looking for problems in users’ lives that we can solve.” But there’s a problem with this. It’s a widely held belief that, if a company can find a problem that is bad enough, people will buy a product that solves it.

That’s often true. But sometimes it isn’t. And when it isn’t true, that’s when really well designed, well intentioned products can fail to find a market."

Read More > 

You Can't Make Good Decisions with Bad Data

I think a critical lesson of the Lean Startup movement is that you have to learn quickly.

The “quickly” part of that lesson can lead to a culture of “good enough.” Your features should be good enough to attract some early adopters. Your design should be good enough to be usable. Your code should be good enough to make your product functional.

While this might drive a lot of perfectionists nuts, I’m all for it. Good enough means that you can spend your time perfecting and polishing only the parts of your product that people care about, and that means a much better eventual experience for your users. It may also mean that you stay in business long enough to deliver that experience.

I think though that there’s one part of your product where the standard for “good enough” is a whole lot higher: Data. Data are different.

You Can’t Make Good Decisions With Bad Data

The most important reason to do good research is that it can keep you from destroying your startup. I’m not being hyperbolic here. Bad data can ruin your product.

Imagine for a moment an a/b testing system that randomly returned the wrong test winner 30% of the time. It would be tough to make decisions based on that information, wouldn’t it? How would you know if you were choosing the right experiment branch?

Qualitative research can be just as bad. I can’t tell you how many founders have spent time and money talking to potential customers and then wondered why nobody used their product. Nine times out of ten, they were talking to the wrong people, asking the wrong questions, or using terrible interview techniques.

I had one person tell me, “bad data are better than no data,” but I strongly disagree here. After all, if I know I don’t have any data, I can go do some research and learn something.

But if I have some bad data, I think I already know the answers. Confirmation bias will make it even harder for me to unlearn that bad information. I’m going to stop looking and start acting on that information, and that may influence all of my product decisions.

If I “know” that all of my users are left handed, I can spend an awful lot of time building and throwing out features for left handed people before realizing that what I got wrong was the original premise. And, of course, that problem is made even worse if I’m not getting good information about how the features are actually performing.

You Have To Keep Doing It

Unlike any given feature or piece of code, collecting data is guaranteed to be part of your process for the life of your startup.

One of the best arguments for building minimum viable products and features is that you might just throw them out once you’ve learned something from them (like that nobody wants what you built).

This isn’t true of collecting data. Obviously you may change the way you collect data or the types of data you collect, but you’re going to keep doing it, because there’s simply no other way to make informed decisions.

Because this is something that you know is absolutely vital to your company, it’s worth getting it right early.

Data Collection Is Not a Mystery

Most of your product development is going to be a mystery. That’s the nature of startups.

You’ve got a new product in a new market, possibly with new technology. You have to do a lot of digging in order to figure out what you should be building. There’s no guide book telling you exactly what features your revolutionary new product should have.

That’s not true of gathering data. There is a ton of useful, pertinent information about the right way to do both qualitative and quantitative research. There are workshops and courses you can take on how to not screw up user interviews. There are coaches you can hire to get you trained in gathering all sorts of data. There are tools you can drop in to help you do a/b testing and funnel tracking. There are blogs you can read written by people who have already made mistakes so that you don’t have to make the same ones. There is a book called Lean Analytics that pretty much lays it out for you.

You don’t have to take advantage of all of these things, but you also don’t have to start from scratch. Taking a little time to learn about the tools and methods already available to you gives you a huge head start.

Good Data Take Less Time Than Bad Data

Here’s the good news: good data actually take less time to collect than bad data. Sure, you may have to do a little bit of upfront research on the right tools and methods, but once you’ve got those down, you’re going to move a hell of a lot faster.

For example, customer development interviews go much more quickly when you’re asking the right questions of the right people. You don’t have to talk to nearly as many users when you know how to not lead them and to interpret their answers well. Observational and usability research becomes much simpler when you know what you’re looking for.

The same is true for quantitative data collection. Your a/b tests won’t seem nearly so random when you’re sure that the information in the system is correct. You won’t have to spend time as much time figuring out what’s going on with your experiments if you trust your graphs.

Good Data Does Not Mean Complete Data

I do want to make one thing perfectly clear: the quest for good data should be more about avoiding bad data than it is about making sure you have every scrap of information available.

If you don’t have all the data, and you know you don’t have all the data, that’s fine. You can always go out and do more research and testing later. You just don’t want to put yourself into the situation where you have to unlearn things later.

You don’t have to have all the answers. You just have to make sure you don’t have any wrong answers. And you do that by setting the bar for “good enough” pretty damn high on your data collection skills.


Like the post? Please share it!

Want more information like this? 


My new book, UX for Lean Startups, will help you learn how to do better qualitative and quantitative research. It also includes tons of tips and tricks for better, faster design. 

To Kill or Not to Kill

After my rant last week about product managers, the excellent Joshua Porter (@bokardo) made a great point about it. He said, “In my own experience the hard part is knowing when to kill something vs. when to give it more breathing room, as sometimes a really new idea can’t really be tested in low fidelity.”

As much as I’d love to send my pageviews soaring by starting a flame war with somebody popular on the internet, I have to admit that he’s 100% right. Killing a feature or product is exceptionally difficult. It’s tough to know when to do it. It’s tough to figure out if you made the right decision. And it’s tough emotionally to let go of something you really thought was going to be great.

First, let’s talk a bit about why you kill products or features. You kill them because they’re not succeeding or because you don't expect them to succeed. That could mean that they’re not getting enough traction or because you’ve determined that they’re never going to turn into an important part of your business. You kill them because the ROI isn’t high enough to justify investing more resources in them. You kill them because they are using resources that would be better spent in other places.

They’re hard to kill precisely because you never know whether they’re just a few days away from taking off and turning into everything you thought they’d be. After all, every successful product went through some period of time before everybody found about it.

So, let’s look at a few questions to ask yourself before killing a product or feature. For the purposes of this post, I'm just going to talk about killing existing features or products. I'll probably address how to decide to kill things before you build them in a future post.

These questions won’t make killing easy, but hopefully, they’ll make it possible.

Why Isn’t Everyone Using It?

There are four reasons people don’t use your product or feature. Yep. That’s right. There may be thousands of reasons that people do use a product, but there are really only four basic reasons that they don’t.

People don’t use your product because:
  • They don’t know about it 
  • It doesn’t solve a problem for them 
  • They don’t understand that it will solve a problem for them 
  • The problem it solves isn’t worth the investment of time, money, or effort
Before you kill your existing product or feature, figure out why it’s not popular. For example, if you’re simply not getting any traffic to your page, it means not very many people know it exists. On the other hand, if you’re getting tons of traffic, but none of it is converting or engaging, then your problem is one of the last three. People are finding your product, but they don't want it or understand it enough to convert.

You can find out if the product solves a serious problem for people by talking to the types of people you expect to have that problem. Develop a persona that represents the sort of person who might suffer from this problem, and interview them about that portion of their lives.

Don’t just ask, “do you suffer from x problem?” Have them tell you stories about their real life experiences in situations where they might have experienced the problem. For example, if you’re testing to see if people need turn by turn navigation on their phones, you might ask them to tell you about the last time they were trying to get somewhere and they got lost. Then you might ask how often that happens. If it’s extremely rare, they probably don’t have the problem you’re trying to solve.

If you’re trying to figure out if they understand the problem your product or feature solves, you can do that by showing them your product or feature (or a mockup or prototype) and asking them to tell you about it. Don’t prompt or prep them. Just show them the product and say, “Tell me what this does. Who do you think it’s for?” You will be shocked by how often your perfectly crafted prose and imagery cause nothing but blank stares.

When determining whether or not your product is worth getting, don’t forget that money isn’t everything to your potential users. Sometimes there are switching costs that they’ll have to deal with or just the cognitive load of learning how to use a new product. I can’t tell you how often I’ve seen people stick with a completely suboptimal solution to a problem, just because that’s what they’re used to.

Regardless of which it is, determining the reason people aren't responding positively to your product will go a long way toward telling you whether to kill it or keep it. 

Who Is Using It?

So, once you’ve determined that there are people who are using your product or who you expect will use it because it solves a serious problem for them at a price they’re willing to pay, it’s time to look at who those people there are and how many of them exist.

A great company with a very engaged group of users recently killed a feature. Unsurprisingly, there was a huge outcry. They got many, many complaints telling them how sad users were that the feature was going away. If they had gone entirely by the comments on the blog post about removing the feature, they would have been justified in thinking that they were making a huge mistake.

Luckily, they didn’t do that.

It turned out that the number of people using the feature was an incredibly small percentage of their user base. More importantly, the people using the feature were not, by and large, paying customers. In other words, a couple percent of very vocal users who didn’t earn a cent for the company were upset by the removal.

While it’s always best to avoid making your users angry, there are certainly users that it’s safer to anger than others. Keeping a feature or product that is disproportionately useful to people who aren’t benefitting your business in some real way means that you have fewer resources to devote to things that might make you some money. 

The other thing to consider here is how many people you might reasonably expect to have use this product if everybody knew about it. Unless there is a huge potential market for your feature or the small market that exists is willing to pay quite a lot to use it, you may want to consider killing it.

Note: for those few people who inevitably write to me and complain that “it’s not all about money,” I would like to point out that it very frequently does have to be about money or you will go out of business. If you want to keep your 10 free users super happy, you go right ahead. I’m going to cater to the large number of folks who pay me. 

And yes, I do understand the difference between long term and short term gains, and I expect my readers do, as well. Assume I'm optimizing for lifetime value here and not simply what makes the most money right this second.

What Is the Actual Cost of Keeping It?

Now that we’ve determined that people are using your product or feature, we should figure out how much it costs to keep your product or feature alive.

Of course, if you’re talking about your whole product, this math is relatively easy. The only hidden cost to keeping your product alive is the opportunity cost of building something else. If you’re working on your current product, you can’t be working on something more promising.

However, if you’re talking about a piece of your overall product, sometimes it can be harder to figure out how much it costs to keep a feature alive. Obviously there is the cost of the engineers or customer support people or sales people, but often they’re working on other things as well, so it’s not clear that cutting any particular feature will really save you any money. If even a few people are using a feature and it’s already built, why not just let it hang around indefinitely?

Well, consider some of these hidden costs to keeping a feature:
  • Bug fixes 
  • Customer support 
  • A more complicated code base 
  • A more complicated user interface (more features means more cognitive load on your new users) 
  • Server and infrastructure costs 
  • Additional work if you decide to do a site redesign or visual refresh
These may seem like small things, and in some cases they are, but don’t ever think that a feature is free just because you no longer are actively building it.

Of course, there's another alternative, which is to continue to iterate on the feature or product. This obviously adds hugely to the expected costs. Let's say that you create a new search feature for your product, and very few people end up using it. The actual cost of that feature needs to include all the iterations and changes that you're willing to try before people start using it or you give up on it. 

What Is the Actual Cost of Killing It?

In a similar vein, sometimes things can cost more to kill than you think. Unhappy users can cause trouble in forums or for support staff.

Of course, I did mention above that it’s sometimes acceptable (and inevitable) to annoy some of your users, but don’t underestimate the work that it will take to keep that unhappiness from spreading throughout your entire community.

There are engineering costs to turning off a feature, as well. Either you need to pull it out of your code base or leave it there to rot. Neither of those options is free, although both can be ultimately cheaper than maintaining the feature.

If you’re killing your whole product, you are often throwing away a huge percentage of your customer base. Just because you’re pivoting doesn’t mean that all of your users will pivot with you.

And the same goes for your employees, if you’re lucky enough to have any. Killing a product or significant feature can be absolutely terrible for morale. Obviously you’re not going to keep a failing product just to make your employees happy, but make sure that you are prepared for the fallout – and possibly resignations - when you do decide to make a major change.

So, Should You Kill It?

In the end, it really comes down to the expected ROI, and the future is notoriously difficult to predict. Good customer development techniques can help you get a clearer idea of the eventual potential of a product or feature that seems to be failing. An honest assessment of real costs can help you determine the investment that you're really making into the product.

But it's an art, not a science. In the end, you're still going to have to make the decision. And it may be the toughest decision you ever make as an entrepreneur. Good luck!

Like the post? Here's what you should do next:


Startups Shouldn't Hire User Researchers

Everybody seems to think I'm a user researcher, which is not strictly true.

It's true, I write a lot about user research, and I've certainly done my share of it over the course of my career. But I don't really consider myself a user researcher. I do enough user research to be extremely effective at being an interaction designer and product manager.

There are lots of actual user researchers with degrees in psychology and specialized training who are doing much more interesting and complicated research than I do. I respect those people. They are scientists in many cases. But I don't think that most of them should work for startups. In fact, I don't think that small startups should ever hire a user researcher just to do research.

Don't get me wrong. User research is critical to startups. How else are they supposed to understand their potential customers and find product market fit?

No, the reason people shouldn't hire people to do their user research is that learning about your customer is the single most important part of your startup. If you're outsourcing that to a person who isn't directly responsible for making critical product decisions, then you are making a horrible mistake.

I see startups do this over and over. They hire a consultant, or even a regular employee, to come in and get to know their users for them. That person goes out and runs a lot of tests and then prepares a report and hands it over to the people in charge of making product decisions. Half the time the product owners ignore the research or fail to understand the real implications of it.

And why wouldn't they? The product managers weren't there for the process of talking to users, so they almost certainly haven't bought into the results. It's really easy to ignore a bunch of bad things somebody else wrote about your idea in a Powerpoint presentation. It's a lot harder to ignore half a dozen real users saying those things to your face and showing you problems that they're having in real life.

The right way to do research in a startup is to have the people who are responsible for making decisions about your product intimately involved in the research itself. That means that product owners and UX people are designing and running the tests. Even the engineers should watch some of the sessions and hear first hand what their users are going through.

The reason I talk so much about user research is that I want you, the entrepreneurs, to learn enough about it so that you can DO IT YOURSELVES. You're welcome to hire people like me, or even real user researchers, to teach you what you need to do. But having somebody else do the research for you is not an option. At least, it's not one that you should use if you're still trying to find product market fit or learn anything actionable about your customers.

Stop thinking of user researchers as people you hire to get to know your users, and start thinking of yourself as a user researcher. At startups, you should all be user researchers, especially if what you really are is a designer or product manager.



Stop Validating Your Product

I talk to a lot of very small companies that are trying to do Customer Development, and the conversations are often the same. The entrepreneur explains that the company is working on a fabulous product, and they want to figure out a) if anybody wants to buy the product and b) if they need to change anything about the product so that more people will buy it.

The entrepreneurs always ask questions like, “How will I know if I have talked to enough people?” and “How do I know if the people who like it are just early adopters?” and “How do I know if I should change the product in response to feedback or if I should just keep trying to find the right market?” The ones who have already been out in the field trying to conduct these interviews all have a sort of glazed, terrified look.

These are all really important questions. I’m going to give you a way to avoid having to ask most of them.

Stop trying to validate your product.

Now, I fully expect a bunch of people to stop reading here and totally miss the point of this post, but for those of you who stick it out, I promise this will make sense in a minute.

The trick is, it is far, far easier to conduct customer development before you have settled on a product or even an idea for a problem.

Why is that? Well, think about products as solutions to problems. Sometimes that “problem” is “I’m sort of bored while I’m waiting for the train” and the unexpected solution is flinging virtual birds at virtual pigs. But often, the problem is something more concrete, and it’s frequently shared by a large group of similar people.

So, instead of focusing on validating a solution, try one of the following techniques.

Validate a Problem

Let go of your preconceptions about how you are going to solve a problem for people and concentrate on first figuring out whether lots of people have a particular problem and what they’re currently doing to solve it.

For example, let’s say that you’ve posited that people have a really hard time finding and making appointments with trustworthy auto mechanics. The mistake you will probably make is to jump right into solving that problem and then going out into the world with some half-baked idea for Yelp meets OpenTable meets AAA and trying to find out whether it solves this problem that you’re not technically sure exists yet.

Instead of doing that, first validate the problem. Get on the phone with lots of different types of people and ask them how they found their mechanics. Talk to them about all of their mechanic-based issues. Find out what causes them the most pain.

Also, this is a good time to narrow down your market. Start with the market “people who have cars and will talk to you,” but quickly start noticing patterns. Do all the busy people have similar problems? What about people who live in cities vs. suburbs? How about people who are new to an area? Try people with special kinds of cars. I’ll bet that they all have very different problems, any of which you might want to solve.

Once you’ve spent time talking to people in various markets with various problems, you’ll come up with all sorts of ideas of how to solve those problems. The great thing is that then you can validate your product idea with people who you already know have a solvable problem.

This is a great way to do things if you have a particular problem yourself, and you want to find out if there are other people like you who have that same problem. By talking to lots of people with the same problem, you’re going to come up with a much better solution than you would if you just solved things for yourself.

Solve a Problem for a Particular Market

A slightly different approach is to pick your market first. Let’s say you have a special affinity for auto mechanics or busy moms or accountants at mid-sized companies.

The trick here is that you’re not going to change your market. You’re going to figure out some massive problem that is being experienced by a large portion of the market, and you’re going to solve it for them.

Your first step is going to be some ethnographic research. You need to really get into the heads of your target market and see what makes them similar and what’s driving them nuts. You’re not going into the research with an idea of the problem you want to solve for them. You’re going to let their needs drive your product decisions.

This is a great method if you happen to have some specific connection with a group or industry. Let’s say you collect porcelain owl figurines. You might desperately want to solve a problem for other porcelain owl aficionados, but you should be open to what problem you want to solve for them. For example, it might be how to get large numbers of high quality porcelain owls. Or it might be ways to contact therapists that deal with severe hoarding issues. Let the user research guide you!

The Easiest Kind of Customer Development

Hopefully you’re noticing a pattern here. The easiest kind of customer development is the kind that you do before you have a very solid product idea.

If you figure out your problems and your market before you come up with an Idea or a Solution or a Product, then when you do build something, you’ve already done a huge amount of the work in figuring out if anybody’s going to use it.

This is really about controlling which variables you’re testing. It’s hard to simultaneously find a problem, a market, and the problems with a real product all at once.

However, once you’ve validated your market and your problem, you can create something that solves that specific problem for that particular market. The beauty of this is that if you build a product for a problem you know exists in a market you know needs it and still nobody uses it, you can be pretty certain that the problem is your product.

Like the post? Follow me on Twitter.

Tiny Tests: User Research You Can Do NOW!

There’s a lot of advice about how to do great user research. I have some pretty strong opinions about it myself.

But, as with exercise, the best kind of research is the kind that you actually DO.

So, in the interests of getting some good feedback from your users right now, I have some suggestions for Tiny Tests. These are types of research that you could do right this second with very little preparation on your part.

What Is a Tiny Test?

Tiny Tests do not take a lot of time. They don’t take a lot of money. All they take is a commitment to learning something from your users today.

Pick a Tiny Test that applies to your product and get out and run one right now. Oh, ok. You can wait until you finish the post.

Unmoderated Tests

Dozens of companies now exist that allow you to run an unmoderated test in a few minutes. I’ve used UserTesting.com many times and gotten some great results really quickly. I’ve also heard good things about Loop11 and several others, so feel free to pick the one that you like best.

What you do is come up with a few tasks that you want to see people perform with your product. When the test is over, you get screen captures of people trying to do those things while they narrate the experience.

Typically, I’ll use remote, unmoderated testing when I want to get some quick insight into whether a new feature is usable and obvious for a brand new user.

For example, if you’ve just added the ability for users to message each other on your site, you can use remote, unmoderated testing to watch people attempt to message somebody. This will help you identify the places where they’re getting lost or confused.

If you’ve done a little recruiting and have a list of users who are willing to participate, you can even ask your own users to be the participants.

And don’t forget, if you don’t have a product, or if you’re looking at other products for inspiration, you can run an unmoderated test on a competitor’s product. This can be a great way to see if a particular implementation of a feature is usable without ever having to write a line of code. It can also be a great way to understand where there might be problems with competing products that you can exploit.

Are you going to get as much in-depth, targeted feedback as you would if you ran a really well designed, in person user test? Probably not. But it’ll take you 10 minutes to set up and 15 minutes to watch each video, so you might actually do this.

Remote Observation

There is something to be said for traveling to visit your users and spending time in their homes or offices. It can be extremely educational. It can also be extremely expensive and time consuming.

Here’s a way to get a lot of value with fewer frequent flyer miles.

Look at the people in your Skype contacts. Find one that doesn’t know much about your product. Ping them. Ask them to do three small tasks on your product while sharing their screen.

Don’t have Skype? Send friends a GoToMeeting or a WebEx link through email.

As with the remote unmoderated testing, this is best for figuring out if something is confusing or hard to do. It’s not very useful for figuring out whether people will like or use new features, because typically the people in your Skype contacts aren’t representative of real users of your product.

The closer the people are to your target market, the better the feedback’s going to be, but almost anybody can tell you if something is hard to use, and that’s information that it would be great if you had right now.

Coffee Shop Guerrilla Testing

Of course, it’s tough to test a mobile app over Skype. You know where it’s easy to test a mobile app? At a coffee shop.

Go outside. Find a Starbucks (other coffee shops are also acceptable if you refuse to go to Starbucks, you insufferable snob). Buy some $5 gift cards. Offer to buy people coffee if they spend 5 minutes looking at your product. Have a few tasks in mind that you want them to perform.

In about an hour, you can watch a dozen people use your app. And if you don’t manage to get any good feedback, at least you can get coffee. But you’ll almost certainly get some good feedback.

This type of feedback is great for telling you if a particular task is hard or confusing. It’s also great for getting first impressions of what an app does or the type of person who might use it.

Five Second Landing Page Testing

Sometimes, all you want to test is a new landing page. What you frequently want to know about a landing page is, “What message is this conveying, and is it conveying it clearly and quickly?” Even the tiniest of tests can seem like overkill for that.

For landing pages, I use UsabilityHub’s Five Second Test. You take a screenshot or mockup of the landing page you want to show. You upload it to the site. You enter a few questions you want people to answer after looking at it.

If the whole setup process takes you more than 5 minutes, you’re doing it wrong, and within a few hours you can have dozens of people look at your landing page and tell you what they think your product does.

This sort of Tiny Test is wonderful for testing several different variations of messages or images that you might put on a landing page. You can get people’s real first impressions of what they think you’re trying to tell them.

CTA Testing

The most important thing to get right on any screen is the Call To Action. After all, you can have the most gorgeously designed images with a wonderfully crafted message, but if people can’t find the damn Buy button, you’re screwed.

But, as with the landing page tests, this is something that takes 5 seconds. Basically, you want to show people a screen and see if they can figure out where they should click. Guerrilla testing works pretty well for this, but even that may be overkill here.

For CTA testing, I often use UsabilityHub’s ClickTest product. Again, you just upload a mock and ask people something like, “Where would you click to purchase the product shown on this page?” or “Where would you go to advance to the next slide?” or whatever CTA you’re testing.

A few hours later, you get a map of where people clicked. If there are clicks all over the place, you’ve got some work to do on your CTA.

The advantage to doing something like this over a/b testing is simply that you can get it set up very quickly with just mockups. You don't have to actually implement anything on your site (or even have a site) in order to test this way. But, if you have enough traffic and a good a/b system already set up, by all means test that way, as well.

What Are You Waiting For?

There you go. Five different options for wildly fast, incredibly cheap feedback on your product. You don’t have to hire a recruiter or write a discussion guide or rent out a usability lab. In a few cases, you don’t even have to interact with a human.

Are they perfect? Do they take the place of more substantial research? Will you be able to get away with avoiding talking to your users forever? No. But they’re easy, and you can do one of them right this second.

So...do one of them right this second!

Like the post? Follow me on Twitter.

Hypothesis Generation vs. Validation

A lot of people ask me what sort of research they should be doing on their products. There are a lot of factors that go into deciding which sort of information you should be getting from users, but it pretty much boils down to a question of “what do you want to learn.”

Today, I’m going to explore one of the many ways you can go about looking at this: Hypothesis Generation vs. Hypothesis Validation. Don’t worry, it’s not as complicated as I’ve made it sound.

What is Hypothesis Generation

In a nutshell, hypothesis generation is what helps you come up with new ideas for what you need to change. Sure, you can do this by sitting around in a room and brainstorming new features, but reaching out and learning from your users is a much faster way of getting the right data.

Imagine you were building a product to help people buy shoes online. Hypothesis generation might include things like:

  • Talking to people who buy shoes online to explore what their problems are
  • Talking to people who don’t buy shoes online to understand why
  • Watching people attempt to buy shoes both online and offline in order to understand what their problems really are rather than what they tell you they are
  • Watching people use your product to figure out if you’ve done anything particularly confusing that is keeping them from buying shoes from you

As you can see, you can do hypothesis generation at any point in the development of your product. For example, before you have any product at all, you need to do research to learn about your potential users’ habits and problems. Once you have a product, you need to do hypothesis generation to understand how people are using your product and what problems you’ve caused.

To be clear, the research itself does not generate hypotheses. YOU do that. The goal is not to just go out and have people tell you exactly what they want and then build it. The goal is to gain an understanding of your users or your product to help you think up clever ideas for what to build next.

Good hypothesis generation almost always involves qualitative research. At some point, you need to observe people or talk to people in order to understand them better.

However, you can sometimes use data mining or other metrics analyzation to begin to generate a hypothesis. For example, you might look at your registration flow and notice a severe drop off half way through. This might give you a clue that you have some sort of user problem half way through your registration process that you might want to look into with some qualitative research.

What is Hypothesis Validation

Hypothesis validation is different. In this case, you already have an idea of what is wrong, and you have an idea of how you might possibly fix it. You now have to go out and do some research to figure out if your assumptions and decisions were correct.

For our fictional shoe-buying product, hypothesis validation might look something like:

  • Standard usability testing on a proposed new purchase flow to see if it goes more smoothly than the old one
  • Showing mockups to people in a particular persona group to see if a proposed new feature appeals to that specific group of people
  • A/B testing of changes to see if a new feature improves purchase conversion

Hypothesis validation also almost always involves some sort of tangible thing that is getting tested. That thing could be anything from a wireframe to a prototype to an actual feature, but there’s something that you’re testing and getting concrete data about.

You can use both quantitative and qualitative data to validate a hypothesis, but you have to choose carefully to make sure you’re testing the right thing. In fact, sometimes a combination of the two is most effective. I’ve got some information on choosing the right type of test in my post Qual vs. Quant: When to Listen and When to Measure.

Types of Research

Why is this distinction between generation and validation important? Because figuring out whether you’re generating hypotheses or validating them is necessary for deciding which type of research you want to do.

Want to understand why nobody is registering for your site? Generate some hypotheses with observational testing of new users. Want to see if the mockups for your new registration flow are likely to improve matters? Validate your hypothesis with straight usability testing of a prototype.

These aren’t the only factors that go into determining the type of research necessary for your stage of product development, but they’re an important part of deciding how to learn from your users.

Like the post? Follow me on Twitter!

User Research You Should Be Doing (but probably aren't)

Startups know they should get out of the building and talk to their customers, but sometimes they’re a little too literal about it. There are tons of ways to get great information from your customers. The trick is knowing which technique answers the questions you have right now.

Sure, you’re doing usability tests and trying to have customer development interviews, but here are a few slightly unusual qualitative user research techniques you should be employing but probably aren’t.

Competitor Usability Testing

Have you ever considered running a user test on a competitor’s site?

This one’s fun because it feels a little sneaky. It also gets you a tremendous amount of great information, since chances are somebody is already making mistakes that you don’t have to make.

For example, when one of my clients, Crave, wanted to build a marketplace for buying and selling collectibles, we spent time watching people using other shopping and selling sites. We learned what people loved and hated about the products they were already using, so we could create a product that incorporated only the good bits.

The result was a buying and selling experience that users preferred to several big name shopping sites that will remain nameless.

Bonus tip: There’s always the temptation to borrow ideas from a big competitor with the excuse, “well, so and so is doing it, and they’re successful, so it must be right!” Guess what? Sometimes other companies are successful for a lot of reasons other than that thing you’re stealing from them. Make sure users like that part of a competitor's product before using it in your own.

Super Targeted Usability Testing

Typically, when conducting usability tests, we’ll run several sessions on an entire product with lots of scenarios and tasks. But often that generates a ton of data that you have to wade through and analyze.

Instead, try doing a few sets of three 10-15 minute tests on a very specific feature. That’s a lot of numbers in a row. How about an example?

When we were building Crave, we wanted to test a particular new feature that we thought users would love. When we actually launched it, we were a little concerned that it would be hard to find, so immediately after launch, we ran three quick, unmoderated user tests with one task.

As we suspected, all three users had some trouble finding the feature. We immediately created a contextual help bubble that guided interested users to the feature. Then we ran three more tests. None of the new users had any problems at all.

The entire process took about three hours, and users regularly tell us how much they like that feature.

Bonus Tip: Using unmoderated user testing services like UserTesting.com and TryMyUI.com (and about a dozen others), make testing like this fast and cheap. You can test, build, deploy, and iterate several times in a single day. If you don’t do continuous deployment, you can use them to test high fidelity prototypes rather than your actual product.

Purely Observational Testing

This type of research is the exact opposite of the last one, because you’re not always testing a very specific part of your interface or a brand new feature.

Sometimes you’re trying to generate ideas for what you could do next that would give you the biggest ROI. For example, you might know that there’s a problem somewhere in your metrics, and you’re trying to understand what pain points are causing the drop off.

Whatever the reason, one of the most enlightening things you can do is a purely observational test. This means sitting down, shutting up, and watching people use your software in whatever way they want to do it.

You don’t give them tasks or scenarios. You just schedule them for a time they’d normally be using your product and arrange to observe them, remotely or in person.

Bonus Tip: Make sure to do this with new users, power users, and occasional users, as well as people who fit in all of your various persona groups. This will give you a fabulous overview of what people are really doing with your product.

Micro-Usability Tests

These are quite different, and some don’t fall neatly into the qualitative testing realm, but they can be quite useful.

Navigation Tests
When we were building Crave, we obviously wanted to make sure that things were incredibly easy to purchase, since that’s where we’d make money.

Since we had wireframes and visual mockups of the screens, we simply loaded them into UsabilityHub’s NavFlow and asked users to show us how they’d purchase something.

After a couple of tests, we knew exactly where in the purchase funnel we needed to improve things before ever even had a real funnel!

Landing Page Tests
Another type of Micro-usability test can help you fine tune your messaging.

Ever run one of those landing page tests where you compare two different messages and see which one results in more conversion? Ever wonder why the winner was the winner?

This is one of those questions that’s not very cost effective to answer with standard usability testing, since what you really want is a couple of minutes of testing with a lot of different users rather than an hour with just a few users.

Luckily, you can post a screen on FiveSecondTest with a few simple questions like, “What does this product do?” and “Who is this product for?” and get extremely cheap feedback about people’s first impression of your landing page.

Now you’ll not only know which version won, but you’ll have a better idea of WHY it won. Different tests that we ran at Crave showed that some messages led people to believe that the site was about “buying and selling” while others led people to believe it was about “sharing” or “meeting people.” And, of course, some messages didn’t mean anything to anybody. We didn’t use those.

Bonus Tip: As with everything, I like running smaller versions of these micro-usability tests iteratively. With the FiveSecondTests, I might run each version of the page with 15 people and then update the messaging until I get a landing page where the vast majority of respondents understand exactly what my product is selling. 


Do These Techniques Replace Usability Testing?

Seriously? Is that even a question? Of course not!

You still need to do regular usability testing and conduct standard customer development interviews. You still need to get out and ask your users questions and have them perform predefined tasks and talk about their problems.

But the next time you want a particular question answered, think a little harder about the best way to answer it and the best tools to use.

Like the post? Follow me on Twitter!

Or check out my presentation on DIY User Research for Startups.

What Does Your User Know?

You and your customer have very different sets of information. This shouldn’t come as a surprise to you, since by now it should be obvious that you are not your user.

Sometimes it can be very hard to distinguish what you know about your product or industry from what your user knows. But it’s important! Making assumptions about your user’s knowledge can lead to products that are impossible for normal people to use.

You Know...

The Details of How Your Product Works

You know all the technical and implementation details of your product. Your user doesn’t even know what those things mean.

One company I worked with had a feature that allowed users to mark off all the items they owned from a list. The engineers knew that, when a user made a selection, that selection was sent to the server in an AJAX request, so the account was kept constantly up to date.

The users didn’t know this. During testing, several users went through, marked off all their selections, and then searched in vain for a Save button. They assumed that they would have to save their work, since they had no idea about the AJAX request that was happening in the background.

The solutions to this were either to allow the users to explicitly save with a button or to give them a tiny amount of feedback while the item was being saved via a very brief wait spinner.

Once we implemented the latter solution, users immediately understood that each click was actually saving the item automatically, and they no longer looked for that Save button.

The take away: Don’t make any assumptions that your users understand what you’re doing for them unless you’re explicitly telling them in some way.

Every Single Feature and Its Purpose

You designed and built every feature in your product, so you know exactly what each of them does and where to find it in your product. Your user knows only what she finds during her time using your product.

One company I worked with had a very useful feature. When I interviewed users about new features they’d like to see, many of them requested the feature, even though it was already in the product! They simply had no idea that it even existed.

The take away: It’s not enough to create fabulous new features. You have to make sure that your features are discoverable by normal users.

Specialized Knowledge About the Product Space

Sometimes we build products to help people do hard things more easily.

Tax preparation software is a fantastic example of this. It is safe to say that most of us who use tax preparation software do not know nearly as much about tax preparation as the people who are building the software. At least, I hope they know more than I do!

The problem arises when we lose touch with exactly which parts of the space the users understand well. We can start to use jargon or terminology that makes perfect sense to us because we hear it all the time. We can design processes that seem completely reasonable if the user already knows the goal.

Unfortunately, this creates complicated, confusing products that assume a much higher level of understanding than the user has.

The take away: When you’re trying to help users accomplish a complicated goal, you need to work even harder to keep the interface simple.

Of course, your customer knows some stuff you don’t know...

How She’s Used to Doing Things

If you’re creating a product that is meant to help a users do something they already do (again, tax preparation or any business software), your goal is to create a generic experience that will satisfy as many people in your core demographic as possible.

But remember, each of those users already does things slightly differently. For example, if you’re helping people who sell things on eBay, you have to understand that each seller already has a process that she follows - from pricing to listing to shipping to dealing with customers.

Asking your users to change too many of their behaviors in order to use your product creates a huge barrier to acceptance.

If you only talk to a few potential customers (or, even worse, none at all), you run the risk of creating something that isn’t broadly usable by all sorts of different users.

The take away: Understanding the variations in user behavior will help you deliver something that is usable by a larger segment of your user base.

Specialized Knowledge About the Product Space

Hang on! Isn’t this something that you have that your user doesn’t have? Actually, yes, but you’re not the only one who knows something special about your product space. Your users may have specialized information about your product that you don’t have.

Imagine you’re creating an application for rating wines (or coffee or chocolate or movies or anything else). You may have very specialized knowledge of how wines are made or how the selling process works or how ratings systems are best implemented.

But your users will likely have very specialized knowledge about various types of wines. They may have quite different opinions about categorization or information display because of this specialized knowledge.

By taking advantage of your users’ knowledge, you can have them help you improve your product without your having to first learn everything there is to know about your product space.

The take away: You don’t need to know everything about your product space, but you should take advantage of the expertise of your users to improve the product and continue learning.

Why Is It Important to Know What Your User Knows?

It’s important to learn what your customer knows so that you’re not creating unnecessary confusion. A product should work with a user’s expectations and understanding rather than demanding the user learn a complicated new way of doing things.

How Do You Learn What Your User Knows?

By listening to them, of course. Having conversations with your users about your product and how they use it will help you to understand exactly what they know and don’t know.

You might even learn something about your own product space, and you’ll definitely learn how to make your product better.

Like the post? There are more like it. You should follow me on Twitter!

Testing Whether Your Users Will Buy

As you all know by now, I’m a huge proponent of qualitative user testing. I think it’s wonderful for learning about your users and product.

But it’s not a panacea. The fact is, there are many questions that qualitative testing either doesn’t answer well or for which qualitative testing isn’t the most efficient solution. I cover some of them in my A Faster Horse post.

The trick is knowing which questions you can answer by listening to your users and which questions need a different methodology.

Unfortunately, one of the most important questions people want answered isn’t particularly well suited to qualitative testing.

If I Build It, Will They Buy?

I get asked a lot whether users will buy a product if the team adds a specific feature. Sadly, I always have to answer, “I have no idea.”

The problem is, people are terrible at predicting their future behavior. Imagine if somebody were to ask you if you were going to buy car this year. Now, for some of you, that answer is almost certainly yes, and for others it’s almost certainly no. But for most of us, the answer is, “it depends on the circumstances.”

For some, the addition of a new feature - say, an electric motor - might be the deciding factor, but for many the decision to buy a car depends on a lot of factors, most of which aren’t controlled by the car manufacturer: the economy, whether a current car breaks down, whether we win the lottery or land that job at Goldman Sachs, etc. There are other factors that are under the control of the car company but aren't related to the feature: maybe the new electric car is not the right size or isn't in our price range or isn't our style.

This is true for smaller purchases too. Can you absolutely answer whether or not you will eat a cookie this week? Unless you never eat cookies (I'm told these people exist), it’s probably not something you give a lot of thought to. If somebody were to ask you in a user study, your answer would be no better than a guess and would possibly even be biased by the simple act of having the question asked.

Admit it, a cookie sounds kind of good right now, doesn’t it?

There are other reasons why qualitative testing isn't great at predicting future behavior, but I'm not going to bore you with them. The fact is, it's just not the most efficient or effective method for answering the question, "If I build it, will they come?"

What Questions Can Qualitative Research Answer Well?

Qualitative research is phenomenal for telling you whether your users can do x. It tells you whether the feature makes sense to them and whether they can complete a given task successfully.


To a smaller extent, it can even tell you whether they are likely to enjoy performing the task, and can certainly tell you if they hate it. (Trust me, run a few user tests on a feature they hate. You'll know.)

This obviously has some effect on whether the user will do x, since they’re a lot more likely to do it, if it isn’t annoying or difficult. But it's really better at predicting the negative case (ie. the user most likely won't use this feature as you're currently building it) than the positive one.

Sometimes qualitative research can also give you marginally useful feedback if your users are extremely likely or unlikely to make a purchase. For example, if you were to show them an interactive prototype with the new feature built into it, you might be able to make a decent judgement based on their immediate reactions if all of your participants were exceptionally excited or incredibly negative about a particular feature.

Unfortunately, this, in my experience, is the exception, rather than the rule. It’s rare that a participant in a study sees a new feature and shrieks with delight or recoils in horror. Although, to be fair, I’ve seen both.

What’s the Best Way to Answer This Question?

Luckily, this is a question that can be pretty effectively answered using quantitative data, even before you build a whole new feature. A lot of companies have had quite a bit of success with adding a “fake” feature or doing a landing page test.

For example, one client who wanted to know their expected purchase conversion rate before they did all the work to integrate purchasing methods and accept credit cards simply added a Buy button to each of their product pages. When a customer clicked the button, he was told that the feature was not quite ready, and the click was registered so that the company could tell how many people were showing a willingness to buy.

By measuring the number of people who thought they were making a commitment to purchase, the client was able to estimate more effectively the number of people who would actually purchase if given the option.

The upshot is that the only really effective way to tell if users will do something is to set up a test and watch what they actually do, and that requires a more quantitative testing approach.

Are There Other Questions I Can’t Answer Qualitatively?

Yep. Lots of them. I’ll probably cover them at some point in the future if people are interested. Feel free to ask about other specific questions in the comments, and I’ll try to let you know what sorts of testing methods work best for answering them.

Enjoy the post? Thanks!
How about following me on Twitter?

The Dangers of Metrics (Only) Driven Product Development

When I first started designing, it was a lot harder to know what I got right. Sure, we ran usability tests, and we looked generally at things like page counts and revenue before and after big redesigns, but it was still tough to know exactly what design changes were making the biggest difference. Everything changed once I started working with companies that made small, iterative design changes and a/b tested the results against specific metrics.

To be clear, not all the designers I know like working in this manner. After all, it's no fun being told that your big change was a failure because it didn't result in a statistically significant increase in revenue or retention. In fact, if you're a designer or a product owner and are required to improve certain metrics, it can sometimes be tempting to cheat a little.

This leads to a problem that I don't think we talk about enough: Metrics (Only) Driven Product Development.

What Is Metrics (Only) Driven Product Development?

Imagine that you work at a store, and your manager has noticed that when the store is busy, the store makes more money. The manager then tells you that your job is to make the store busier - that's your metric that you need to improve.

You have several options for improving your metric. You could:
  • Improve the quality of the shopping experience so that people who are already in the store want to stay longer
  • Offer more merchandise so that people find more things they want to buy
  • Advertise widely to try to attract more people into the store
  • Sell everything at half off
  • Remove several cash registers in order to make checking out take longer, which should increase the number of people in the store at a time, since it will take people longer to get out
  • Hire people to come hang out in the store
As you can see, all of the above would very likely improve the metric you were supposed to improve. They would all ensure that, for awhile at least, the store was quite busy. However, some are significantly better for the overall health of the store than others.



The same thing happens all the time when designing products. If your assigned goal is to increase the number of active users, there are lots of different design changes you could make, but not all of them will be equally effective for improving the actual goal, which is probably increasing the number of people who use the product and generate revenue.

How Does This Happen?

I think the biggest reason that this happens is that people fixate on metrics without understanding the reason behind the numbers. Designers and product owners are then pressured to move a number that represents a particular metric rather than focusing on improving the product.

One company I talked with had this problem with acquisition. The person who was responsible for acquiring new customers was simply given a budget and told to get as many users as possible for that amount of money. Unfortunately, the users that were cheapest to acquire were the least likely to spend money on the site. If, instead of trying to maximize the number of users, he had concentrated on maximizing the number of users who were likely to spend money, he would have acquired fewer people and missed his metric, but he would have increased revenue.

Another company had a design problem. They wanted to redesign their Invite a Friend feature to encourage people to invite more friends. Unfortunately, the "most effective" method of getting people to invite friends was to forcibly spam users' Facebook feeds and make it easy for users for accidentally invite everybody in their address books. While this resulted in more invitations sent, it also vastly increased the number of unhappy customers and decreased the percentage of invitations that were accepted. It also caused the company to be banned from Facebook and put on the spam list of several ISPs. It sure improved that invitation metric, though.

How Should You Avoid It?

There are three ways to avoid this problem, and you should use all of them.

Make sure that you're measuring the right metric.

If you care about revenue (and you should), measure revenue. If you care about retention, measure retention. If you care about page views, you're probably doing something wrong.

Unfortunately, it can be difficult to immediately see the impact of a particular design change on things like revenue and retention, which makes it tempting to use substitutes for the important number.

If you are using a substitute - for example, if you're using something like "customers returning once" as a shorthand for "becoming an active customer" make sure that the link is actually causal. In other words, if you can increase the number of people who come back once, make sure that that really does lead to an increase in people who become an active customer.

Make sure that you're not gaming the metrics.

Paying people to come back to your site may result in more returning customers, but it doesn't necessarily result in more customers paying you. If you cut your prices in half, you may end up selling twice as many items, but you're not making any more money. Make sure that, if you're moving a metric, you understand the second (and third and nth) order effects of whatever change improved the metrics.

Make your customer experience better.

This may seem obvious, but pissing off your customers is a terrible long term strategy, even if it briefly moves a metric. On the other hand, improving your customers' overall experience leads to happy, contented customers who stick around and continue to pay for your service. Over time, that's going to improve all of your metrics.

And always remember, metrics are just shorthand for real customer behaviors that are important to your business. They are a tool to help you understand your product, not a goal to be met at any cost. 

Like the post? Follow me on Twitter.

Your User's Computer Sucks

I often tell people that “you are not your user.” I’m an interaction designer. It’s part of my job description. What I should probably also be reminding people is that your computer is not your user’s computer. Nor is your internet connection, monitor, environment, or a lot of other things.

Why is this important? These things mean that, aside from making your product usable in a lab setting or in your office, it’s also got to work well in the standard environment of your user.

So, ok, if you build tools for lean startups, you can probably ignore most of this post. Your users all have dual core machines (probably more than one), fiber internet connections, 24 inch flat screen monitors, and a cubicle or office in which to use those things. They probably also have a smart phone that never leaves their hand and a comparable set up at home so that they are never more than a few inches from enough computer power to get them burned as witches in some parts of the world.

But the rest of you probably build products for teens or busy families or doctors or construction workers or business travelers or, you know, the vast majority of humanity that doesn’t use a computer for a living. And you need to not only understand your users; you need to understand your user’s computing environment.

I do a lot of in-home/office studies as well as remote usability testing. This means that, not only do I get to see real users with my products, I get to see them use them on their computers, and I’ve seen this over and over.

Here are a few examples of how your user’s environment really matters:

Your Computer is Faster than Your User’s


Interactions that take a split second on my machine sitting right next to the server may take two or three seconds on an old computer half way around the world. This doesn’t seem like much, but it has a pretty big impact.

When an interaction takes a split second, I don’t need to plan for any intermediate feedback (a spinner, a progress bar, a disabled button, etc.). When an interaction takes three seconds, I really, really do need one of those things, or else I’m going to get repeated button presses and confused users who don’t know whether anything is happening. Of course, interactions that are annoyingly slow on my computer are going to be completely intolerable to a lot of my users.

You Have More Computers than Your User Does


I have a laptop and a smartphone at home that are all my own, and I use them constantly.  The other day, I asked a user why she chose to use a product late at night. She explained that she had school aged kids who needed the computer in the afternoons, and during the day she was typically out of the house. Her only computer time was a few minutes after the family was all in bed.

Many products are used by multiple people during the day on the same computer, sometimes at the same time. Having limited time on a shared computer creates all sorts of design implications for things like privacy and the need to be able to interrupt and resume tasks.

Your Monitor is Bigger than Your User’s Monitor


Of course, then there’s monitor size. Many people have declared the death of “the fold” because people don’t mind scrolling a bit for interesting information, but I still see products with really important calls to action that don’t show up on screens smaller than a bay window.

Guess what? I’ll scroll to read the second half of a blog post, but I’m not going on a damn treasure hunt for your Buy button. If I don’t find it quickly, there are a whole lot of other sites that will sell me what I want. If your target audience accesses your site on laptops or smartphones or Etch A Sketches, figure it out and design accordingly.

Look, “getting out of the building” isn’t just another way of saying “chatting with customers.” It means understanding customers and how they use your product. A big part of how people use your product is dictated by the environment in which they use it. So make sure that you don’t only understand who is using your product and how they are using it. Learn WHERE they’re using it and on what sort of equipment. It can make a huge difference.

Like the post? Follow me on Twitter!

Love Thy User: 5 Tips for Dealing with Tough Customers

Sometimes we build products for ourselves, but most of the time our target market is somebody completely different. It can cause all sorts of problems when we’re asking questions and observing people completely different from ourselves. Sometimes, and this can be tough to admit, we don’t really like our users very much.

Maybe you’re not like this. Maybe you’ve never had a difficult set of users who constantly yell and scream about their needs and how they’re not being met regardless of what you do for them. Maybe you’ve never spent time building a brand new feature designed to make your users happy only to have them shrug and say, “oh, that’s not what we wanted at all.” Maybe you’ve never had a passionate community of early adopters all grumpy because their favorite suggestions aren’t being followed to the letter. But trust me, the rest of us have.

The problem is, because most of our users are so different from us, their behavior can be extremely hard to understand or predict. On many occasions, this has led people to ignore their customers or neglect to include them in the development process.

I understand this impulse. I really do. It can be tough to include somebody that you see as irrational or hard to deal with in your decision making process.

But here’s a news flash. That irrational, difficult, whiny, impossible to understand person who is always complaining? Suck it up, cupcake, and include them in the conversation. They’re paying your salary, and if you ignore them for long enough, they’re likely to stop doing it.

Here are a few ways to make it easier to get feedback from difficult groups of customers.

Keep it one on one

When you’ve got a group of people, all of whom seem hell bent on complaining about how your product is ruining their lives, don’t put them in a room together.

A lot of companies like to establish customer advisory panels or customer forums and the like, where they can get feedback from a lot of people at once. These are fine when the conversation can be kept civil, but they can quickly turn into an angry mob as the group forms a giant echo chamber of hate.

Keeping the conversation one on one allows you to spend more time with each person and understand what’s really upsetting him or her.

Keep the conversation in person (or on the phone)

TALK to your customers. Sure, forums, suggestion boxes, and other written forms of communication are useful tools, but when you’ve got angry or unhappy customers, written communication is just too easy to get wrong. Many people who are more than happy to dash off an angry screed in a forum will calm down immediately once they get a real human being to explain their problems to.

It also makes it quick to ask follow up questions and really get to the root of the matter without inadvertently making the customer angrier.

Identify the root cause

When people are angry, they’re not always great about expressing what is upsetting them. What sounds like, “You are ruining my life,” might actually mean, “There is a very specific bug that you’ve been ignoring that is affecting the way I run my business.”

Remember, there’s almost certainly a nugget of truth at the center of the craziest assertion. The best way to find out that truth is to get to the root cause of the problem as non-confrontationally as possible.

Try asking the users to walk you through exactly what is causing their problems. Make sure to ask when and how the problems first started. If they give you a laundry list of complaints, try to get them to prioritize. Often, there is one core issue that is causing the anger, and dozens of other minor complaints that just add fuel to the fire.

Also, once you know the real problem, don’t assume you understand the solution. Remember, you caused the problem in the first place. Ask users to help you come up with solutions. I’m not saying you should just do whatever they ask, but make sure you check possible fixes with them before implementation to make sure that you’re actually fixing the root cause of the problem.

Maintain trust

One important lesson you must learn is that it’s much easier to maintain trust than to reestablish it after it’s been lost.

One company with which I worked had a passionate group of power users who all seemed to be conspiracy theorists. Whenever something went wrong or a bug brought down the site, they would immediately begin concocting wild stories about nefarious future plans that the company had to institute various Orwellian policies.

The company could never understand why the users didn’t believe it when the company explained that there were no nefarious plans in the works. Many at the company wrote the users off as crazy.

The problem was that the company had, in the past, issued blanket statements denying policy changes and then gone ahead with those exact policy changes. After that had happened once, there was absolutely no reason for the users to believe anything the company had to say, even when the company was telling the truth.

Once lost, trust was almost impossible to regain, and many of those passionate power users moved on to other products.

Actually fix their damn problems

Ok, understanding that your users aren’t just raving lunatics is a great first step. Connecting with them and identifying their problems is even better. Now you actually have to FIX THINGS. The best communication in the world isn’t going to help you if you continue to ignore the things that your customers are complaining about.

I know, I know. You’re working on the next release or a hot new feature or an integration with a partner or any one of a number of other things that your customer may or may not care about one bit. But now that you you’ve asked them, you know a lot about what your customer DOES care about, and you should work on those things too. Right now.

In conclusion

Look, I’m not saying that the customer is always right. Sometimes people really are crazy, or maybe they’re just not right for your particular product. That’s fine. You’re going to lose those customers regardless of what you do.

But sometimes the fact that you don’t understand your customers isn’t their fault. It’s yours. Learn to listen to your customers rather than writing them off as angry cranks. Within a very short while, you’ll find that they don’t sound so crazy, after all.

Like the post? Follow me on Twitter!

When To Get Help With User Research

I don't spend a lot of time on this blog telling you why you should hire me to talk to your customers. In fact, the vast majority of the posts are meant to make it more possible for you to talk to your customers without hiring somebody like me. It's not that I don't like working. It's just that I think that anybody who is responsible for making decisions about products should know how to learn from users on their own. It results in better products for all of us.

Product owners need to be involved in customer research for a lot reasons. Reasons like:
  • You're more likely to believe the results if you participated in the research.
  • You're more likely to understand the relative importance of customer problems if you observed the problems happening.
  • You will come up with more comprehensive solutions to problems when you understand the context in which they're happening.
  • It's far too easy to ignore a report written up by a usability consultant, it's incredibly easy to forget to watch the testing videos.
  • If you do it yourself, all of the lessons you've learned will stay within the company, long after a consultant has gone on to other projects.
That said, I'm about to tell you why you may need to hire somebody like me. For a little while at least.

When I talk about customer research or customer development or learning from customers, I really mean quite a lot of different techniques. Sure, there are general best practices around talking to customers, tips for improving your research skills, and important things you should avoid, but there are also things like picking the right testing method or tool that you almost certainly have no experience with. You need to know what is the most important thing for you to do right now.

Do you know when it would be helpful to do a card sort? A journal study? A contextual inquiry? Do you know when it's fine to do a remote usability study vs when you should really run one in person? How about when your product will benefit from using an online tool like usertesting.com or fivesecondtest, and when something like that isn't useful? Do you know what sort of testing to do in order to find out why specific metrics are lower than you'd like? Do you know when you should start your visual design and when you need to concentrate on usability? Do you know how many people to talk to in order to answer a specific question? Do you know at what points in the development cycle talking to users is critical and when it's a waste of time? Do you know how to take several hours of free form user conversations and turn it into a small number of features or bug fixes that can be communicated to your engineering team?

If you answered, "of course I know that" to all of those questions, then move along. You almost certainly have no use for somebody like me to come in and help you out. If you answered, "I'm going to learn the answer to all of those questions," then I wish you good luck on your journey of discovery. I'll warn you though. There are more questions just like those.

If, on the other hand, you said, "I don't know the answer to a lot of those questions, but I wish somebody could help me understand the small subset of them that matter to me, as a product owner, so that I could get on with the business of building a great product," then you might want to give me (or somebody like me) a call.

Because it's true that there is a huge amount to know about talking to your users. But it's also true that, at any given stage in your product development, you probably only need to be concerned with only a little bit of it. And, it's also true that figuring out which bit of it you need to know can be really hard to do without help. That's where people like me come in handy. We can help you figure out what to do next, and then we can help you learn how to do what you need to do next.

But be careful. If you're a lean startup, you probably don't want to pay us to actually do what you need to do next. For all the reasons I mentioned above, that's still your job.

Interested in this sort of service? Learn more about Users Know here.

Want to read more posts on how to do this stuff yourself? Follow me on Twitter!

Shut up, and Show Me Something

I admit it. Quite often on this blog I give you long lists of fairly hard things to do. I ask you to change your whole approach to design or product management or customer interviewing or analyzing data. But not today. Today, I share with you one simple thing that is easy to remember and will transform your entire approach to customer research.

Ok, maybe it's not quite that cool, but it really will help you communicate with your customers better. Are you ready? Here it is:

Never have a conversation with a user (or potential user) where you don't show them something.

That seems simple enough, right? But why on earth should you do it, and what could you possibly show them?

Reasons for Communicating with Customers

Let's back up for just a moment. The main reasons that people generally talk with a user are:
  • To get information from them - what they like, what they don't like, what's confusing, why they're not buying things, etc.
  • To give them information - here are the features of the product, here's how to fix your problem, we swear it's a feature and not a bug, etc.
  • To sell them something - whatever it is that sales people do...besides drinking heavily
All of these things are much easier to do when you're looking at visual aids.

Getting Information from Users

Let's perform a thought experience. Without thinking about it, name three things you hate about doing your taxes. Were you able to do it? Of course you were. If you can't think of three things you hate about doing your taxes, either you're not paying attention, or your hiding all of your money in an offshore account in the Caymans. But are they really the three worst things?

Probably not. They're just the three things that you happen to think about when put on the spot. Next tax season, you'll be doing your taxes and think to yourself, "Oh right, THAT thing! I hate that thing! I wish I'd thought of that when I was asked for three things I hate." And you most likely would have thought of it if you'd been going through your tax preparation software when I asked.

Sure, you can just ask users what they like and dislike about your product, but you will get much better information if you're both looking at the product together. Even better, ask them to perform some tasks or just use the product while you watch. This not only jogs the user's memory about all the little annoying things that they're sort of used to, but you can also observe all the things that they don't even notice or are too embarrassed to mention they're having trouble with.

Giving Information to Users

Think about getting slightly complicated directions from somebody verbally. Now imagine that they're showing you the route on a map while they talk to you. Which do you remember best? What about if you have a gps screen showing you turn by turn directions in the moment? Even better? You bet.

Looking at things together helps your customer understand the information you're delivering much better than just talking.

Selling Users Something

I'm not a sales person, so I'm not going to spend any time on this one, but I'm a hell of a lot more likely to buy something that I have actually seen in action (and liked) than something that has been extensively described to me. If anybody has any examples, pro or con, of how showing people a product impacts sales, share them in the comments.

But What Should I Show Them?

Good question. You should show them whatever it is you want to get feedback on, give information about, or sell. The last two are pretty self-explanatory, but the first one can be tricky. Let's look at it more closely.
If you have a product and want to know what people think about it and how they're using it, look at the product with the user. Easy enough.

If you have an idea for a product, and you want to get feedback on it, show sketches or interactive wireframes. The closer to a real product it looks, the better the feedback will be, but there are probably times when you just want to show a bunch of quick sketches or a few different visual designs in order to get quick impressions.

But what if all you have is a vague idea of a problem you want to solve? You're not even at the sketch or wireframe stage yet, but you know that you want to solve some problems for a particular group of people. What do you show them then? Easy, does the user have some other method or product that they're using to solve the problem currently? Watch them use that! Are they not currently solving the problem at all? Have them show you how far along they got in solving the problem before they eventually gave up.

Here's an example. A client of mine once wanted to create a solution for people who sell things online. Now, there are two potential markets for this:
  1. People who are currently selling things online through other services
  2. People who want to sell things online but don't, for some reason
For the first market, we needed to watch people selling things online the way they currently do, in order to figure out what their problems and pain points were. For the second market, we needed to watch the way people sold things offline and then watch them try to use competitors to understand what was keeping them offline.

Could we have just asked people those questions? Sure, but we wouldn't have gotten information that was nearly as detailed or complete. In every observation, there were times when the user would get to some point in the process and say something like, "Oh, right. That thing there. That drives me nuts." There was also a point where we noticed that the customer was jumping through crazy hoops to complete a task, but they were so used to it working badly that they didn't even imagine that it could possibly work better. Later, when we showed them our possible solutions to problems they didn't even know existed, they were thrilled.

But Laura, Why Don't You Ever Include Visual Aids In Your Posts?

I was hoping you wouldn't notice that, frankly.

Know of a case when you wouldn't want to show somebody something? Argue with me about it in the comments!

Like the post? Follow me on Twitter!

Nobody is Thinking About Your Product

When you're working at a startup it can be all-consuming. You can forget everything else in your life pretty easily when you're neck deep in valuations and minimum viable products and customer acquisition and a million other things that need your attention. You tend think about your product every waking minute.

That's why it can be such a shock to realize that nobody else is thinking about your product. Well, ok, unless you're Apple, but there's clearly some kind of weird mind control thing going on there. In general, when you have a new product, you're incredibly lucky if you're getting more than a few minutes of attention from anybody but your most passionate early adopters.

Why is it important to realize this? It's important, because it has a really big impact on how you design your product and connect with your users.

Make Everything More Discoverable

You know exactly where in the user interface to go to do every task that can be completed with your product (I hope!). Other people, especially new users, don't even know that most of your features exist. This means that it's just as important to design for discoverability as it is to design for usability. But how are they different?

Let's do a quick thought exercise. Imagine somebody hands you a featureless metal box. You might look at it for a minute or two. If it's particularly attractive, you might admire it, but you're probably not going to spend a lot of time with it. Now imagine that the box has $10,000 dollars inside of it. You will probably spend a lot more time figuring out to get it open, yes?

Your product is like that box that is hiding money. If people don't discover very quickly that it provides something valuable to them, they're not going to spend much time figuring out how to use it. You need to help people understand immediately that your product has features they really, really want. That's discoverability.
You also need to make it pretty easy to actual learn how to use those features, once they've decided to dig into the product a bit. That's usability. For bonus points, you can make the whole process interesting and engaging so that people actually enjoy discovering features and using your product. That's fun. 

Key Take Away: Users are not going to spend any time learning to use your product if they don't immediately understand what's in it for them. Make it easy for them to figure out what features exist and why they're useful.

Remind People You Exist

This is going to sound obvious, but people have lots of things to do every day. What with their own jobs and families and checking Facebook and standing in line at the Apple store for whatever is coming out next, their schedules are pretty full. If you're going to fit into that schedule, you can't just sit around and wait for them to come back to you.

Remember, they're not thinking about you. Ever. That's why you have to contact them regularly via email or Facebook or Twitter or post cards or sky writing or whatever you think will get the attention of the people you're targeting. Think that you can put off writing your welcome emails or your mass notification system? The longer you put it off, the more early users you're going to lose because they didn't think to come back to your product the next day. 

Key Take Away: Assume every single one of your users forgot about the existence of your product five seconds after closing it. If you want them to come back, you need to remind them.

Design for Distraction

You know what you never see in a traditional user test? The seventeen million other things that your users are doing while using your product.

See, even when people are thinking about your product, they're not thinking about only your product. They're thinking about the phone that's ringing and when they have to pick up the kids from soccer and what they're going to make for dinner and the fact that their boss wants a TPS report finished before 5.

Things like shopping carts that time out or registration forms that need to be filled out from scratch if you make a small error or login redirects that don't send you back where you wanted to go are all poisonous to the distracted user. They dramatically increase the number of people who are going to simply give up on your product halfway through.

That's why it's important to make sure that you're actually watching people use your product in their natural environments whenever possible so that you can understand the kinds of interruptions that you need to plan for. It's also critical to make sure that, if somebody were to get called away from your product in the middle of a task (which they will), that they can easily come back and finish that task without having to start over from scratch. 

Key Take Away: People have other things going on in their lives, so make sure that your product allows for interruption, inattention, and distraction.

Make It Addictive

In the way that heroin addicts always think about heroin and WoW addicts always think about Wow and Apple fans always think about new Apple products, if you can design something addictive people will think about your product more. The problem is, making things addictive is harder than just throwing in a few simple game mechanics or copying whatever WoW or Apple does.

There are a lot of good blog posts on how to make your product stickier, but some of the common themes include:
  • creating social bonds with other users (Joe wants to be your friend!)
  • having time sensitive tasks that require users to return at certain intervals (Log in in the next 15 minutes to get this great deal!)
  • providing incentives and achievements for regular use (You unlocked the Foozle Badge!)
  • providing competition with other users (You are now the mayor of the Scranton, PA Taco Bell!)
  • creating a sense of disaster if they don't return (Your fake crops will die!)
  • offering quality content that the user can't get anywhere else and that updates daily (Learn about the five things in your pantry that could KILL YOU!)
The important thing to do here is to pick styles of addiction that fit with your product. Your tax preparation software probably doesn't need a leaderboard, but a good, weekly blog on ways to save tax-free money for retirement might be a draw. 

Key Take Away: Provide reasons for your users to want to come back daily by using things like game mechanics, social pressure, or new content, but make sure that the features fit comfortably with your product.

Cultivate the People Who DO Think about Your Product

If you're lucky and you work really hard to get people's attention, everything I've said isn't going to apply to a tiny group of early adopters. These are the people who will think about your product all the time and want to be heavily involved in its growth and improvement. Use those people! Talk to them. Learn from them. Get them to evangelize your product to other people just like them. Give them jobs, or let them monitor your forums and answer questions for other users. They will love that they're contributing to something they care about, and your product will improve as a result.

Also, if you ignore them, they'll most likely stop thinking about your product, and go think about somebody else's product. 

Key Take Away: Understanding the few people who are deeply attached to your product can help you understand and improve the features that may make it appealing to other, less dedicated users.

But Most Importantly...

You need to internalize the fact that, even once they've visited your site or downloaded your app or become a registered user of your product, the vast majority of people simply aren't thinking about you or your product. At all. This means that a big part of your job is not so much about countering loathing or dislike as it is about countering total indifference.

So, get out there and make them remember you exist.

Like the post? There are more like it. Follow me on Twitter!

When Talking to Customers Isn't Enough

A few weeks ago, I was talking to the head of a small company about next steps. His company had a product with many happy, paying customers, and he wanted to know what to do next to make his current customers happier and attract some new ones.

The company had already made a good start. They had done surveys of current users asking the standard questions like, "How disappointed would you be if you could no longer use this product." They'd even surveyed current users about what new features they would like to see. The problem was, the happy customers couldn't think of anything else they wanted, and while the slightly less happy customers wanted some new features, there was no general consensus on one big, missing piece or fantastic new idea. So, the CEO wanted to know what to do next.

I believe this can be a problem with the "go out and talk to your customers" solution to product development. We can get so focused on talking to customers that we forget that it's not always the best way to figure out what to do next.

Observe Your Customers

Customers lie. They don't always mean to lie, but it often ends up that way. It's especially problematic when you ask people to explain how they currently use your product. Sometimes they forget parts of their process, or they don't even realize that they're doing certain things because it's all become so routine. Also, they tend to explain their process of using your product from start to finish, as if they weren't doing seven other things while trying to use your product. This can give you a totally skewed vision of what people are actually doing with your product.

What's the solution? Go out and watch them. Sit with them in their offices or homes and observe their real behavior. Most importantly, watch what they do immediately before and after they use your product.

I was talking with somebody who used to work for a marketplace that allows people to buy and sell products directly to each other. While observing users, her team noticed that, while people had very little trouble using the marketplace itself, the sellers spent a huge amount of time arranging shipping of the items. In fact, the shipping process took so much time that it limited the number of items they could list. By integrating with a shipping company to help customers print labels and schedule pickups, the company increased the number of items that could be listed which increased revenue.

Why didn't users ask for that? Well, the customers had a particular way of doing things. They thought of the marketplace as a place where they could buy and sell things, not as a product that helped them mail things. They had another solution that helped them mail things, and they didn't know that there was a better way to do that until they were presented with it.

Another critical thing you can learn by watching people is the environment in which your product is being used.  In one study I conducted, I was watching people process payroll. When asked how they processed payroll, customers could easily explain all the steps they went through. However, when I sat down beside them and watched, I realized that it wasn't nearly that simple. Not a single person got through the process uninterrupted. Phones rang. Coworkers stopped by to ask questions. Information was missing and had to be hunted down. Bosses needed tasks performed immediately. What they had described as a quick, linear process actually happened in fits and starts and could take place over a day or two.

Were they lying when they described their experiences? Not intentionally. They weren't asked to describe everything that could possibly happen while processing payroll, and they probably couldn't have answered that question if I'd asked it, since the interruptions varied wildly depending on the day and the office. In the end, the observations helped make the product more tolerant of this working style by allowing people to save state, skip areas where they didn't have the right information, and easily track what had already been done and what was still pending.

Connect With People Who Were Almost Your Customers

Don't forget, there's another really important group of people out there: people who were almost your customers. For every one person who signs up for your service or converts to a paying customer, there are lots of people who took a look or maybe used a free trial and then decided not to pull the trigger. A great way to build your customer base is to figure out why they made that decision.

The company I mentioned at the beginning of the post had a perfect audience for this. They offered a one month free trial, which meant that they had information about people who used the product, saw exactly what they had to offer, and then chose not to become customers. Maybe they didn't convert because the product was lacking a key feature. Maybe they didn't understand how to use it. Maybe it didn't do what they expected it to from reading the description on the website. These are all totally fixable problems, but you can't fix them until you know what they are.

Let me just head off the inevitable criticism of this approach right now. I am not advocating that you listen to every single thing that your almost-customers ask for and start building those features immediately. That would be insane. What I am suggesting is that you listen to the reasons that they give for not using your product and then analyze the data for patterns. Are lots of people saying that the product didn't do what they expected? Maybe the problem is that your marketing materials are deceptive. Are they complaining that it didn't do what they wanted? Find out what they wanted to do and what they're currently using to do it. How you incorporate their feedback is up to you, but you can't respond to feedback unless you're asking for it.

Of course, non-customers can be a little harder to connect with than customers, and they do tend to be less likely than customers to allow you to come hang around their offices all day and watch them work. Starting with a survey or an email asking to interview them on the phone can get you lots of good information about what is keeping them from becoming customers. Once you've built a rapport, some of them might even let you come watch them use the product.

Take Another Look at Customer Data

Not all companies have the ability to collect extensive customer data, but if you do, you may not be taking full advantage of it. For example, companies often fail to figure out the difference between the sorts of people who do become customers and those who don't.

Is your product only being purchased by companies with fewer than five employees? If so, that may be a signal to increase your marketing efforts to small companies while decreasing your spend on advertising to larger ones. Are your customers disproportionately mothers or college students or left handed circus performers? If so, start connecting with people who fit that profile to see what they think of your product and whether it solves a particular need for them that you might not have known anything about.

Or, the difference could be based on behavior rather than demographics. For example, if you have a freemium model or a free trial period, you should be looking at the behavior leading up to a user converting to paid or abandoning the product.

One client I worked with created a huge model showing all the different behaviors of users to try to understand which behaviors were most likely to lead to higher revenue and retention. Once we knew that users who explored a particular part of the product in the first five minutes were most likely to pay us, we could start experimenting with what would happen if we emphasized that part of the product early. Once we found that people who went down a different path abandoned the product, we could study that particular flow and find out if we were unintentionally confusing people or driving them away.

So Should I Still Talk To Customers?

Of course you should! Staying in constant contact with customers is vital to understanding your market and keeping people happy. It's just not always enough. If you feel like talking to customers has left you at a dead end or you want to get a perspective from somebody who isn't already a customer, give some of these alternate methods a try. You might be surprised at what you learn.

Like the post? Follow me on Twitter, please.

You should also check out some of my other posts on user research and customer development:

Your Users Are Doing Something Surprising

This post is for all of you lucky enough to have a product with real users. Way back before you had users, or even a product, you probably went through a process to figure out what you should build. During that process you may have written user stories and work flows that described, in various levels of detail, how your users would perform each expected task. But you know who didn’t read your user stories? That’s right: your users.

The result? Somewhere out there, a whole lot of your users are doing something totally unexpected with your product.

Ok, maybe it’s obvious that sometimes users do the unexpected. Maybe you expected your SMS-based social network to help people find out where their friends are in real time, but instead celebrities started using it to market directly to their fans. Maybe you created a product designed to help bands promote themselves, but instead ended up with a social network where people hook up with their high school sweethearts. Or, although this seems extremely unlikely, maybe you had an MMO but people just used it to share photos.

Whatever they’re doing, it’s something you didn’t expect, but it’s very important that you learn what it is as soon as possible!

Why Should You Care?

There are three excellent reasons for you to know what your customers are actually doing with your product:
  • So you know if you are missing an opportunity to pivot your product or marketing
  • So you know if you are missing an important feature
  • So you don’t accidentally destroy a commonly used workaround or "unplanned feature"

Missing a Huge Opportunity

The first reason is pretty well demonstrated by the examples I’ve already given. Flickr could have gone on being an MMO that its customers also used for sharing photos, but the company really took off once they jettisoned the majority of their product and started concentrating on the part that customers found most valuable. Figuring out the surprising thing your customers are doing with your product allows you to decide if you would be better off eliminating 90% of your product and concentrating on the 10% people actually want.

Missing a huge business opportunity doesn’t only apply to pivoting your product. You could also be missing a huge marketing opportunity. One of the benefits to understanding more about what your users are doing with your product is that you can start to segment them better. For example, maybe a small group of your users are using your product in a very particular way. Now, maybe those users are all alike in some important way. If this is true, maybe you should be marketing this new use of your product to other people like them or emphasizing this use in your marketing materials.

Missing a Feature

Sometimes understanding what your users are trying to do with your product can tell you a lot about what you should be helping them do with your product. We all know that users are terrible at predicting their future behavior, but this isn’t their future behavior we’re talking about. It’s their current behavior. Your customers want to do something with your product so badly that they’re going out of their way to come up with clever ways to do it on their own.

Again, to use Twitter as an example, think of retweets and hashtags. Neither of those was built into the original product, but users saw missing functionality and provided it.

Destroying a Workaround or Unplanned Feature

The third reason is a little less obvious, but it’s an outstanding way to make yourself loathed by some of your most passionate users. Sometimes even a small change can completely destroy a workaround that people were using to get your product to behave the way they want it to.

Some form of this happens virtually every time a new browser update gets released. Each time a new version of IE or Firefox comes out, thousands of web developers tear their hair out trying to figure out why the website that looked so lovely in the previous version starts looking like it was designed by Salvador Dali. The reason is often because some hack that they used to achieve a particular effect is no longer supported in the upgraded version.

Now, you can argue with me all you want about how web developers shouldn’t be using browser hacks to get their CSS right or that this only happens to bad developers, but let me remind you that this is just an example. And, in this example, the web developers represent your paying customers. If you ruin the way that they’re used to doing something, they become a lot more likely to go find somebody else who will let them do things the way they want.

How Can You Find Out What They’re Doing?

This is one of those cases where there is absolutely nothing more beneficial than watching your current users interacting with your product in their own environment. You will never get this information in usability tests with people who have not used your product, and you will not get nearly as much good information by simply interviewing current customers without observation.

Let me be perfectly clear: the best way to get this information is to spend time observing your users in their natural habitat, whether that’s their home, office, car, or wherever they use your product. You should schedule appointments with them at times when they would naturally be using your product anyway, and you should ask them to think out loud as you sit quietly in a corner and take notes. After they’ve spent a good amount of time with your product, you can interview them about their experiences and follow up on any workarounds or odd behaviors that you saw. Note: Be respectful, please! Don’t make your customer feel like a freak for doing something you didn’t expect.

If you absolutely can’t be in the same room as your user, you can also get good information by observing remotely using screen sharing and webcams, but in person is really ideal. One of the big reasons that simply interviewing them about their habits won't work as well is that they don't realize they're doing anything unexpected, so they can't tell you where their behavior differs from what you expected.

You should, of course, also be recording as many facts as possible about usage stats so that you have a statistical overview of which parts of your product are being used most, but this is one of those cases where you can learn things you never expected just by watching customers in person. It's often tough to spot these things in data. In fact, you may not even think to gather data on this particular behavior, since you didn't expect it in the first place!

What Should You Do About It?

Obviously, this doesn’t mean that you can never change anything that a customer is using as a workaround unless you’re totally pivoting your whole product. That would be insane. What it does mean is that you should be aware of what your actual users are doing so that you can improve the experience for them, or at least not make it worse by removing a beloved workaround or unplanned feature.

Remember, if your customers want so very badly to use your product for a particular purpose, even if it’s not the purpose you originally intended, you might want to make it easier for them to do it.

Like this post?  Do some of the following:

Pain Driven Design

I have a problem with both User Centered Design and Customer Driven Development. This may come as something of a shock to people who read my blog, since I’m constantly giving advice on better ways to talk to users, analyze data, and improve customer development efforts.

The problem I have with UCD and CDD is not the methods. The problem is that people so often misunderstand them. People hear “user centered” and think, for some insane reason, that we are encouraging them to turn their entire design process over to the user. They hear “listen to your customer” and think that we want them to blindly accept every ridiculous feature request made by somebody with a credit card and an opinion.

Guess what? I rarely speak for entire communities of people, but I think I can safely say that nobody in the user centered design or customer driven development business are asking that of anybody. If they are, they’re idiots and shouldn’t be listened to anyway.

Unfortunately, a lot of us have debunked this myth by explaining that we don’t actually think that customers can predict future behavior (even their own) or that they’re going to have some grand design vision for your product. We just think that customers are great at talking about their problems and pain points, and that those are good things for you and your designers to know when you create a new feature or product.

I’m suggesting that we come up with a new name that will be harder (or at least more fun) to misinterpret: Pain Driven Design.

What Is Pain Driven Design (PDD)?

The PDD methodology requires that, before you start design for a product or feature, you need to figure out what is causing pain for your users and potential users. The desired outcome of PDD is to make that pain go away by some brilliant method of your devising. You then check to make sure you made their pain go away without causing them any more pain.

Is There a Clever Analogy?

There is! Imagine you’re a doctor. You want to help people feel better. The first thing you need to do is to talk to patients who come to see you. Now, of course, you’re not going to ask them what disease they have and how they think you should treat it. You’re going to ask them, “Where does it hurt?” You’re probably going to also ask them a lot of other questions about how they normally feel, what their medical history is, what their family is like, and other things like that. You’ll probably also, if you’re a good doctor, examine them, double check their reported symptoms, and check for symptoms they may not even know they have.

Then, when you have figured out what is causing them pain, you will determine a good course of treatment that is likely to work based on your knowledge of various diseases, your extensive medical training, other work in the field, and how the patient reacts to treatments. You will then closely monitor their progress and adjust your approach as necessary.

Pain Driven Design is a lot like that. You will talk to your users and potential users and find out what causes them pain when they are trying to solve a problem. You will interview them about their habits, likes, and dislikes. You will observe them using the product or competitors' products and look for commonly appearing symptoms. You will then decide how to go about curing their pain. And, of course, you will closely monitor all of your users to see how they respond to your treatment. Since you have a large number of users, and there aren’t any pesky rules against human experimentation in this context, you will run tests to see which treatment performs best.

Does It Work Before I Have a Product?

Sure it does! Presumably, your eventual product will solve somebody’s problem, yes? Maybe their problem is that it is too hard to find a great restaurant while traveling or that they are sort of bored while waiting for the train. OK, those don’t seem like big problems, but they are problems nonetheless and should have solutions.

Since you don’t have a product yet, you need to figure out how people are currently solving this problem. Are they using a similar product? A completely different one? Are they simply suffering in silence without even knowing that their lives would be infinitely better if this problem would go away?

You can find these things out by asking people about how they’re dealing with their problems and what the pain points are with their current solutions (or non-solutions). You can learn more about their pain by watching them suffer through it. Don’t worry, it’s not so bad to watch them suffer, because you know your product will help them!

What If I Already Have a Product?

It still works! You see, no matter how much you love your product, unless it’s perfect, it’s causing pain to somebody. I’m sure it’s not on purpose. You’re not a monster. But something about your product is confusing or hard to use, and it’s driving at least one of your customers nuts.

Again, examine them. Find out when they feel the most pain while using your product. Watch brand new people use your product to see how much pain it takes to learn. Watch old customers use your product to figure out what weird workarounds they've created to avoid the pain that you’re causing them.

Find all the pain points. Then come up with devastatingly clever ways to fix them.

What If My Product Is Disruptive?

It's still solving a problem, right? Even if it's solving that problem in a completely novel way or solving it for a new group of users, presumably if people are going to adopt the product, the product will be solving a particular problem for them.

That’s great . Even if people have never seen anything like your product, you can get a huge amount of information by talking to users about how they currently solve their problems as well as their general environment for problem solving. And once your disruptive product has been launched, chances are it’s causing some people some pain, so you should observe people interacting with it to learn where the pain points are.

What If My Customers Try to Tell Me How to Fix Their Problems?

Well, I suppose you could plug your ears and scream loudly so that you won’t be in danger of hearing them talk about their solutions. Or you could listen to their solutions and then politely follow up to make sure you understand the underlying pain that they’re trying to cure. Frankly, I prefer the latter approach, but it’s up to you.

One interesting thing that I’ve found in my many, many years of listening to customers is that sometimes the customers are right about the solution. I know, crazy! I mean, we've been assured by hundreds of people that listening to customers' solutions is completely useless and that they're always wrong! Guess what? They're not. This doesn’t mean you should take their word as gospel, but I can't imagine that people within your company have a patent on coming up with the right solutions to customer problems. Just because an idea for a solution comes from a user doesn’t automatically make it useless or wrong, even if the anti-UCD crowd seems to think so.

How Will Pain Driven Development Be Misinterpreted?

Who can say? I sort of hope somebody reads only the title of this post and writes a scathing retort about how I’m a horrible person for advocating that designers be subjected to pain until they come up with better designs (note: they shouldn’t, except in certain very specific cases, and they know who they are). Or maybe they’ll dissect my doctor/patient analogy and point out all the ways in which it’s flawed (note: there are 17 ways. See if you can spot them all!) and thereby conclude that, since my analogy isn’t perfect, my methodology must also be crap.

But I hope a few people will say, “Hey, that Pain Driven Design methodology is just a catchy name for understanding our customers’ problems so that we can come up with better solutions!” And more importantly, I hope a lot of people will say, “You know, that Pain Driven Design sounds like a great idea, and we should try it in our organization!”

Like the post? You could leave me a message in the comments, tweet about this post, read some of my other posts, tweet about them, or hire me to help you learn how to find your customers' pain points! Those are all valid options.

You should also follow me on Twitter.

5 Mistakes People Make Analyzing Qualitative Data

My last blog post was about common mistakes that people make when analyzing quantitative data, such as you might get from multivariate testing or business metrics. Today I’d like to talk about the mistakes people make when analyzing and using qualitative data.

I’m a big proponent of using both qualitative and quantitative data, but I have to admit that qualitative feedback can be a challenge. Unlike a product funnel or a revenue graph, qualitative data can be messy and open ended, which makes it particularly tough to interpret.

For the purposes of this post, qualitative information is generated by the following types of activities:
  • Usability tests
  • Contextual Inquiries
  • Customer interviews
  • Open ended survey questions (ie. What do you like most/least about the product?)

Insisting on Too Large a Sample

With almost every new client, somebody questions how many people we need for a usability test “to get significant results.” Now, if you read my last post, you may be surprised to hear me say that you shouldn’t be going for statistical significance here. I prefer to run usability tests and contextual inquiries with around five participants. Of course, I prefer running tests iteratively, but that’s another blog post.

Analyzing the data from a qualitative test or even just reading through essay-type answers in surveys takes a lot longer per customer than running experiments in a funnel or looking at analytics and revenue graphs. You get severely diminishing returns from each extra hour you spend asking people the same questions and listening to their answers.

Here’s an example from a test I ran. The customer wanted to know all the different pain points in their product so that they could make one big sweep toward the end of the development cycle to fix all the problems. Against my better judgment, we spent a full two weeks running sessions, complete with a moderator, observers, a lab, and all the other attendant costs of running a big test. The problem was that we found a major problem in the first session that prevented the vast majority of participants from ever finding an entire section of the interface. Since this problem couldn’t be fixed before moving on to the rest of the sessions, we couldn’t actually test a huge portion of the product and had to come back to it later, anyway.

The Fix: Run small, iterative tests to generate a manageable amount of data. If you’re working on improving a particular part of your product or considering adding a new feature, do a quick batch of interviews with four or five people. Then, immediately address the biggest problems that you find. Once you’re done, run another test to find the problems that were being masked by the larger problems. Keep doing this until your product is perfect (ie. forever). It’s faster, cheaper, and more immediately actionable than giant, statistically significant qualitative tests, and you will eventually find more issues with the same amount of testing time.

It’s also MUCH easier to pick out a few major problems from five hours of testing than it is to find dozens of different problems from dozens of hours of testing. In the end though, you’ll find more problems with the iterative approach.


Extrapolating From Too Small a Sample

I always do this don’t I? Say one thing, and then immediately warn you not to go too far in the opposite direction. The thing is, I get really tired of running five person tests and having a product owner only show up for one session and then go off and address whatever problems s/he saw during that one hour. One or two participants aren’t enough to really get a sense of the pattern of problems in your product.

Besides, I have this little rule of thumb I’ve developed for studies. No matter how great your screener or recruiter, on average for every 10 participants you schedule, one will be a no-show, one will be some sort of statistical outlier (intelligence, computer savvy…something), and one will be completely insane. If the product owner happens to show up only for one of the last two types, their perception of the product’s problems will be totally skewed.

I had one product where we interviewed ten people over the course of two tests. Nine of the ten people were wildly confused by the product, but one, who I swear was a ringer, nailed all the tasks in record time. Guess which session the product manager showed up for? Yeah.

The Fix: As the person making the decisions about what changes you should make in your product, you should be attending all or at least most of your user interview sessions, even if you’re not running them yourself. You should also be looking directly at all of your survey data, not just skimming it or reading a high level report. Honestly, if you’re the one making decisions about product direction, then you are the one who most benefits from listening to your users. If you’re not paying attention to the results, then the testing is really just a waste of time.

Look at all your data before drawing conclusions. I mean it.

Trying to Answer Specific Questions

Qualitative data is very bad at answering specific questions like “Which landing page will people like better?” or “How much will people pay for this?” What it’s great for is generating hypotheses that can then be tested with quantitative means.

In more than one test, I’ve had clients ask me to test various different images to use on landing pages to see which one was most appealing. I always explain that they’re better off just doing a split test to see which one does best, but sometimes they insist. Unfortunately, these sorts of preference differences are often very subtle. Since people are not making the decisions consciously, it's very hard for them to explain why they prefer one thing over another. We always end up getting a lot of people trying to rationalize why they something, and I rarely trust the results.

The Fix: Use qualitative data to generate hypotheses that you then test quantitatively OR to find major problems in your interface. Don’t try to use qualitative data to get a definitive answer to questions about expected user preferences.

Ignoring Inconvenient Results

Because qualitative testing doesn’t generate hard numbers, it’s easy to let confirmation bias sneak into the analysis. While it might be tough to argue with “this registration flow generated 12% more paying customers than the other one,” it’s pretty easy to discount problems observed in user sessions.

I dealt with a particularly resistant product owner who had an excuse for every single participant’s struggles with the product. One was unusually stupid. Another just didn’t understand the task. Another actually understood it but was, for some reason, actively screwing with us. This went on and on while every single participant had the same problems over and over. Also, the discussion guide, which the product owner and everyone on the team had originally thought was perfectly fair, suddenly became wildly biased and the tasks were judged to be impossible. The problem couldn’t possibly have been with the product!

The Fix: If you are finding fault with all of the participants or the moderator or the questions or the survey, it’s time to get somebody neutral into the room to help determine what is biasing the results. Hint: it’s almost certainly you.

Remember, your customers, moderator, and test participants don’t have a stake in making your product seem worse than it is. You, however, may have an emotional stake in making it better than it actually is. Make sure you’re not ignoring results just because they’re not what you want to hear.

Not Acting on the Data

Why would you even bother to run test if you’re not going to pay attention to the results? I mean, tests aren’t free. Even running surveys has an associated cost, since you’re interrupting what your user is doing to ask them to help you out. And yet, so many clients do exactly this.

One client I worked with wanted to set up a system where they ran tests every week. They wanted to have a constant stream of users and potential users coming in all the time so that they could stay in contact with their users. I thought this was a fantastic idea, and so I started bringing people in for them. Unfortunately, after a few months, people began to complain that they were hearing the same problems over and over again.

I explained that they were going to continue to hear the same problems over and over again until they fixed the problems. I gave them a list of the major issues that their current and new users were facing. Every once in awhile, if I complained loudly enough, they would fix one of the easier problems, and unsurprisingly these changes always positively affected their metrics. And yet, it was always a struggle to get the results from the tests incorporated into the product. I eventually stopped running tests and told them that I would be happy to come back and start again as soon as they had addressed some of the major problems.

The Fix: This one should be simple. If you’re going to spend all that time and money generating data, you should act on the results.

I want your feedback!

Have you had problems interpreting or using your qualitative data, or do you have stories about people in your company who have? Please, share them in the comments section!

Want more? Follow me on Twitter!

Also, if your company is currently working on getting feedback from users, I’d love to hear more about what you are doing and what you’d like to be doing better. Please take this short survey!