Venture Hacks

Thanks to KISSmetrics for supporting our interview with Sean Ellis. If you want an intro to KISSmetrics, send me an email. I’ll put you in touch if there’s a fit. Thanks. – Nivi

Hiten Shah from KISSmetrics recently sat down with me to explain how to use their product, survey.io [update: this has moved to PMFsurvey], to measure product/market fit and find the “best grass” in your product. You may know Hiten from his Crazy Egg days.

survey.io is a free survey tool that helps you implement some of Sean Ellis’ techniques to get to fit. KISSmetrics actually built survey.io with Sean. Unfortunately, there hasn’t been great documentation for survey.io besides Sean’s launch post. Until now.

SlideShare: How to measure product/market fit
Audio: Interview with chapters (for iPod, iPhone, iTunes)
Audio: Interview without chapters (MP3, works anywhere)
Transcript with highlights: Below

Prerequisites

You’ll get more out of this interview if you also read:

1. An example survey from survey.io.
2. Our interview with Sean Ellis.

Outline

Here’s an outline and transcript of the interview. The interview and transcript are about 19 minutes long so we’ve highlighted some of the juicy bits to get you started.

1. survey.io: Before product/market fit
2. survey.io measures fit
3. How did you discover the product?
4. How would you feel if you could no longer use the product?
5. What would you likely use as an alternative if product were no longer available?
6. What is the primary benefit that you have received from the product?
7. survey.io is open-ended
8. The flock will always find the best grass
9. You don’t need survey.io to find the best grass
10. Have you recommended the product to anyone?
11. What type of person do you think would benefit most from the product?
12. How can we improve the product to better meet your needs?
13. survey.io is more powerful with filtering
14. Would it be okay if we followed up by email to request a clarification to one or more of your responses?
15. Upcoming survey.io features
16. Get qualitative feedback before fit
17. Must-have % by industry
18. Ask the must-have question

Transcript

Nivi: Hi there! This is Nivi from Venture Hacks, and I’m here with Hiten Shah from KISSmetrics.

First of all, I want to thank them for sponsoring our interview with Sean Ellis and making it available, free, to you guys.

We’re going to talk a little bit about what KISSmetrics does and how it ties into the things that we discussed in the interview with Sean. And Sean is actually an advisor to KISSmetrics. Is that right?

Hiten Shah: Yes, that’s right.

survey.io: Before product/market fit

Nivi: Cool. The interview with Sean was broken into two parts — before product/market fit, and then after product/market fit, where you do the things to prepare for growth.

Why don’t we talk a little bit, first of all, about what you guys have and do for the before product/market fit stage?

Hiten: Sure.

Nivi: Take it away!

Hiten: Yeah, I will! First of all, I want to thank Venture Hacks. We’re happy to support it and thank you guys for bringing interviews like Sean and Eric and all those guys to a bigger audience. I think you guys are doing a great job of that, and that’s why we’re pretty happy to sponsor the podcasts.

In terms of pre-product/market fit we’re following a lot of these practices in our own business, so we’re eating our own dog food, so to speak — everyone says that we’re actually really doing it.

survey.io measures fit

Hiten: So that’s why we built survey.io with Sean Ellis, and the idea of it was, Sean’s got a bunch of questions he asks. The key question is: how disappointed would you be if a product name, say, Google didn’t exist? There are a bunch of key questions there. I’m sure it’s gone over in the interview, previously, in the first part.

But basically, there wasn’t a template that had that question on it. So we worked with Sean Ellis to come up with a simple product, a simple tool that was free that people can use to assess whether they have hit product/market fit, and that’s what we did with survey.io.

So anyone can go into survey.io and basically send off the survey to their users and get an idea of whether they have a product/market fit.

1. How did you discover the product?

Nivi: So the main questions are: How did you discover the app?

2. How would you feel if you could no longer use the product?

Nivi: How would you feel if you could no longer use the app — which is the key product/market fit question — very disappointed, somewhat disappointed, or not disappointed at all?

3. What would you likely use as an alternative if the product were no longer available?

Nivi: What would you use as an alternative?

What’s the thinking behind that? Do you have any thoughts on that?

Hiten: Yeah. It helps you understand how you fit against competitors, or what space you fit into. You might think that you’re a customer feedback tool, while customers might weigh you as a customer support tool. So that question is actually pretty telling in helping you understand, from your customer’s perspective, what other products they would compare your product with. So I think actually that’s a pretty important question.

Nivi: Have you ever seen any surprises on that, just to flesh it out, either with you guys or other guys?

Hiten: Yes, actually I have. In a few early questionnaires about the KISSmetrics product, an older version of it, people actually put us up against optimization tools like A/B testing tools and things like that, and we didn’t have any functionality around that. So I think that was an interesting insight. Maybe it was just the fact that the analytics space is pretty crowded and there are a lot of tools in it and it’s kind of a nebulous definition, like what is analytics?

4. What is the primary benefit that you have received from the product?

Nivi: The next question in survey.io is: what is the primary benefit that you’ve received from the product?

What’s the thinking behind that?

Hiten: The thinking around that is, basically, if you understand what benefits people see your product for and you filter it by people that would be very disappointed if your product didn’t exist, you can get a really good idea of what passionate customers consider the benefit of your product — and in their own words. So you can use that to change your messaging, and kind of A/B test a bunch of things around landing pages and things like that and see if you can have a higher conversion rate based on the messaging you have there.

survey.io is open-ended

Hiten: I’m thinking that this interview with Sean has probably had a strong impact on getting people to understand what these questions are all about, much more. And that’s what I’m hoping happens because I get a lot of questions about all of these things, as well: What does each question mean? What do I do with these answers?

And that’s actually a typical problem with most survey tools, as well. If you notice, in this survey there aren’t any questions that are very least, most, least, and have radio buttons when you go there, because you’re basically making your customer think a lot with those types of questions. With these open-ended questions people are just typing in whatever they want to say, and then it’s up to you as the person giving the survey out, to analyze the results. So, that’s a big difference with this survey and the questions that are asked.

The flock will always find the best grass

Nivi: Right. Going back to the question: what is the primary benefit you’ve received from the product? Sean quotes Vinod Khosla a lot that, “the flock always finds the best grass.” I didn’t actually understand what he ever meant by that until the interview where he… What he really means is, look at the must-have users and see where in the product they consider the grass to be — they’ve found the grass. Before that, I didn’t even really understand what Vinod meant by it. I guess that’s what he means by it. Right?

You don’t need survey.io to find the best grass

Hiten: Yeah. Let me give a different perspective. I think that’s absolutely correct, but let’s say you don’t know about product/market fit, and you don’t know about giving out this survey or asking that question because you’ve never seen Sean’s stuff. Someone who has a lot of customers, and only some of them are addicted to the product, what they typically do is go look in their logs or look in their user database and see who’s logging in the most or see who sends support requests or is contacting the company the most, and they would consider those people the flock that’s going towards being very passionate about the product.

So I think there are number of ways you can determine this type of stuff. So to me, what Vinod Khosla said feels like there are multiple ways you can determine this. This seems to be a very strong and efficient way compared to how people used to do it.

In some of our own products, in the past, we would just do it based on: Oh! This guy logs in every day. Or, this guy keeps bugging us and wants features, and at the end of every email says, “Oh, I love your product!” We’re actually getting that with our latest product that we have, and those are the people who typically you try to cater to and things like that. This puts a little more science behind it.

Nivi: Right. And this tells you why they love the product.

Hiten: Absolutely.

5. Have you recommended the product to anyone?

Nivi: The next question is: Have you recommended the product to anyone? And if so, please explain how you described it.

What’s the thinking behind that?

Hiten: This goes back to something called Net Promoter Score.

These questions are all developed by Sean, but this question is interesting because when I first saw it, it reminded me of Net Promoter Score. Net Promoter Score is a methodology that a lot of Fortune 500 companies have implemented, where they ask people: how likely are you to recommend this product? And they have a scoring system based on the responses.

Nivi: And it’s a scale of 1 to 10.

Hiten: Yeah, it’s a scale. Exactly. It’s not a simple question. It makes people think. But they have scoring around it, so I don’t know exactly how the scoring is. But if the average people say 3 to 4, and then you make some changes, you want to see those people start saying 5 to 6, or whatever it may be. And then you group… It’s not a gauge….

Nivi: Do you take cohorts of those?

Hiten: Yeah, you take cohorts of those people and try to improve them over time. And there’s a whole methodology around it. It’s pretty complex. Another part of it is….

The key thing about this question that people don’t get at first glance is that they think it’s really asking them if they’d recommend, and trying to make them recommend to people, or trying to suggest they should recommend it, and it’s actually not about that, and neither is Net Promoter Score. All it is, is that there is some psychology around if you’re willing to say you’d recommend it. You would be bucketed into more of a passionate customer. It doesn’t mean necessarily that you’ve ever recommended it, or that you would, but it’s more just to solicit how satisfied customers are with your product.

So this question doesn’t come out of NPS, because Sean didn’t even know about Net Promoter Score, I believe, when he created it, but it’s got the same kind of feeling towards it. So it’s just basically another identifier of people who are passionate.

But the real thing about asking them the yes or no question is when you ask them to describe how they would describe it to a friend, because in this case it’s an open-ended field and they’re typing in the way that they would describe it, which is basically another gauge, a different sort of angle on: what benefit have you received from the product? You’re having them actually help you figure out what your messaging should be, quite frankly.

Nivi: Yeah, and that is, literally the positioning of the product, the position it holds in the user’s mind, and that’s how he would describe the product to someone else.

Hiten: Absolutely. Yep, it’s really powerful.

6. What type of person do you think would benefit most from the product?

Nivi: Just this next quick question: what type of person do you think would benefit most from the product? Question 6.

Hiten: Yeah, I actually have a term that relates to this question a little bit, but my term is data porn. Some things are just data porn. This doesn’t fall into this category, but it just reminds me of that because I get a lot of enjoyment out of this question when I look at results, because people will describe it in all sorts of ways. You’ll hear people say: my best friend would best use it.

You know, depending on what type of product, obviously. I think this question is important because it helps you understand what your target market should be, potentially. Right?

Nivi: Right.

Hiten: And a lot of times people won’t even say it’s someone like them. They might say that it’s someone different than them, and that could give you some insight into, based on your current messaging, what people really think the type of person who would be attracted to your product would be.

So, I haven’t learned as much from this question, but it’s always fun to look at, because it’s another one of customers saying something.

Nivi: Just thinking about the interview with Sean, which you haven’t heard yet, I think this could be helpful, in particular, when the product is very horizontal and you’re trying to figure out which vertical in that group is really the one that you’re going to go after, potentially. For example, Sean talked a little bit about how they initially positioned the product in a feature-based way so as not to narrow down the use case.

Hiten: Yup.

Nivi: So here you’re basically asking them the title of the person who’s going to use the product, or what the use case is.

Hiten: Yeah. Yeah.

Nivi: So I think it can help you narrow down the positioning and the market that you’re going to go after.

Hiten: That’s a great point.

7. How can we improve the product to better meet your needs?

Nivi: How can we improve the application or product to better meet your needs? Question 7.

Hiten: That’s always a great question. People tell you…

This one’s interesting. I think this one gets the most quantity, especially with passionate people. They want to tell you exactly how you can improve their product.

If you haven’t asked this question to your customers, I don’t understand why not, because they’re going to give you all kinds of insights into little tidbits they don’t like. I’ve seen people answer this question and go into: oh, the button should be moved; or the UI is like this; or even try to get into really descriptive stuff.

survey.io is more powerful with filtering

So it’s just a cool question, and again, all these questions are really powerful when you start filtering. That’s why actually we don’t have an export feature that’s public on this thing, but a tidbit is that if you just add a /export to your results, you’ll get an export of the data.

Nivi: OK, nice. Yeah, and the filtering would actually be super interesting. The first segment filter is by must-have, nice-to-have, and I-don’t-care users.

Going back to Question 7: how can we improve the product? Again, you haven’t heard the interview yet, but what Sean talks about is going to the nice-to-have users and figuring out how you can improve the product for them and turn them into must-haves.

Hiten: Yup.

Nivi: So he gave the PayPal example, where some of the nice-to-have users may have still been using the mobile to mobile payment. You don’t care about those guys because you’ve seen like 1,000x increase on desktop to desktop payment. But of those nice-to-have users who are using the desktop to desktop payments, you might want to know how they want to see the product improve; for example, better security or if it had my address book in there already or something like that.

Hiten: Yup.

8. Would it be okay if we followed up by email to request a clarification to one or more of your responses?

And then: would it be OK if we followed up by email to request a clarification? Question 8. What’s the thinking behind that?

Hiten: These are people that you have… you know what they answered, and you can segment into different groups and you can follow up with them with follow-up surveys and things like that. It’s always good to get email addresses.

What we’ve noticed, on the weakest products, the ones that aren’t as sticky, maybe are more consumer oriented, we’ll still see 50% of the people giving an email address. And you might not even have the email address of these people because one of the things we let you do with the survey is embed it on the page, and it might not be a logged-in user so you might not even know what their email address is.

So now all of a sudden you’ve got these people, you’ve got their email address, and you’ve got all these answers associated with that email address. So that just becomes really powerful to follow up with them or, back to the previous question, if you add a feature that they want, you can go notify them and let them know you added it because they wanted it. Customers get delighted by that. I don’t see why every company isn’t trying to delight their customers — it’s one of those things.

Nivi: Right. And when you’re in the trying to get to product/market fit stage, getting a list of potential, early evangelists is probably at the top of your to-do list. Right?

Hiten: Absolutely. Yeah, we don’t even let you edit the questions yet. That’s a feature coming out. We’re building a much deeper product out of this. We have some ideas around it. We’re still working with Sean on it.

Upcoming survey.io features

Nivi: What’s coming in this? The segmentation would be cool.

Hiten: Yeah. It’s been a very interesting conversation at our own company about what we should do about this. One thing is we want to, obviously, have this same thing for free, but we’ve also got some ideas on how to get qualitative feedback from customers in a much leaner way where you almost ask them one question at a time. You might build up some of these answers over time. So we’re working on embeddable surveys and things like that, to do that, and this is stuff we really haven’t seen anywhere else.

Also there are things like, if you know someone’s username or email address, you could already pass in to the survey so that you don’t have to ask for it. And then you can associate a specific customer with the response, and stuff like that, which SurveyMonkey and other tools currently make a little bit tedious to do, or basically impossible in some cases.

We don’t want to compete with SurveyMonkey or the survey tools, so we’ve really thought long and hard about what we want to do about this. We will be launching something, probably by mid-to-end-of January that’s a whole revamp on this tool. It makes it so, hopefully, you don’t have to copy and paste the questions into other tools. So editing, filtering — we’re really focused on embeddable surveys. These are just some of the pains that we’ve noticed about getting feedback from our own users.

If you ask them one question within the interface, it’s really powerful because you can get a lot more context. So in our system, if they’re looking at an analytics report, we might want to ask them: is this report useful to you? And if they say yes, say great, thanks! We love you. If they say no, we can pop-up a form field and ask how we can improve it for them. Those are the kinds of more mini-surveys and mini-questions that we don’t really see a product online that does that, so we’re going to move the product in that direction.

We’ll always have this specific survey for free though, and add some of that stuff. And this is all about….

Get qualitative feedback before fit

We feel that before product/market fit, what’s really important is those early customers and getting as much feedback as you can that’s qualitative from them, and quantitative metrics aren’t as important as the qualitative metrics at that time.

Must-have % by industry

I’m amazed by some of the percentages I see on the disappointed score that’s segmented by industry. In general, trends-wise, if it’s a SaaS sort of product, like a web app, I tend to see a higher score on the percentage of people that would be very disappointed.

Nivi: So that would actually be a great little study for you guys to publish. Right?

Hiten: Yeah. And another thing, on the opposite end, is that we notice score in the 20% to 30% for consumer applications. Typically they’re very low. People wouldn’t say that about Facebook, necessarily, but a lot of people would say that about more trivial consumer sites. I would say that you would probably see a lower score on something like YouTube, where it’s very passive and people aren’t necessarily very passionate about it and there are a lot of alternatives and there isn’t really some sticky glue that makes people really want to come back. So those scores are usually under 40%, while SaaS apps I’ve seen as high as 80%

Nivi: Well that makes sense, especially if you’re going to filter by… SaaS guys are already paying for the product; they’re probably going to consider it to be a must-have.

Hiten: Yeah, and even some of those freemium businesses though. Think about the emotional attachment to a free product you have if you’re constantly using it. You’re going to be disappointed if it doesn’t exist. Right? So I think those are the trends that we see, overall.

Ask the must-have question

Nivi: The key question here, if you’re going to walk away with one thing from survey.io, is that you should probably survey the people who actually use the product and ask them how disappointed would they be if they could not use the product anymore.

Hiten: Yup! Absolutely.

To be continued in part two of our interview with Hiten Shah, where we’ll talk about their product called KISSmetrics, a new tool for optimizing funnels.

Music: Squarepusher