Steve Blank: Customer Development 2.0 “Why Accountants Don’t Run Startups”

Steve‘s talk is a must-watch. I’ve watched it twice now.

Kent Beck: Beyond agile programming

Kent Beck is a natural speaker, brilliant, and humble to boot. Fans of agile software development and extreme programming will especially enjoy this talk. At least watch until he talks about scratching goats — that’s my favorite part.

There were many other excellent talks at the conference — see them all here. Now I’m looking forward to the Startup School conference, coming later this year.

But my favorite comment is from Rob May, founder of Backupify and AngelList power broker:

“In the early days of Backupify (when it was still called Lifestreambackup), not only was I worried that someone would steal the idea and execute it better, but I was also embarrassed by the quality of the product when I demo’ed it.  But ultimately, some early users loved it and that kept us going.  By releasing an early buggy minimum viable product, we got a pretty big lead on everyone else.  Now we have a bunch of competitors, but with more than 60K users and 30 terabytes of data backed up, we are pretty far ahead in terms of traction. Being open early worked well for us.”

By the way, Backupify recently raised money from General Catalyst and First Round Capital.

Thanks to KISSmetrics for supporting our interview with Sean Ellis. If you want an intro to KISSmetrics, send me an email. I’ll put you in touch if there’s a fit. Thanks. – Nivi

Hiten Shah from KISSmetrics recently sat down with me to discuss how to optimize funnels with their upcoming analytics tool, KISSmetrics.

You may know Hiten from his Crazy Egg days and survey.io, which Hiten and I discussed in How to measure product/market fit with survey.io.

SlideShare: How to optimize your web apps with KISSmetrics
Audio: Interview with chapters (for iPod, iPhone, iTunes)
Audio: Interview without chapters (MP3, play anywhere)
Transcript: See below

Prerequisites

You’ll get more out of this interview if you also read:

1. Our interview with Sean Ellis parts 1 and 2.
2. Our previous interview with Hiten: How to measure product/market fit with survey.io.

Outline

Here’s an outline and transcript of the interview. The interview and transcript are about 23 minutes long and we’ve highlighted some of the juicier bits for you.

1. After fit, prepare for growth
2. KISSmetrics helps you optimize your funnel
3. KISSmetrics helps with all optimization steps
4. First user experience isn’t necessarily the paid user experience
5. KISSmetrics is most valuable after fit
6. How Cloudfire uses KISSmetrics during fit
7. Startups aren’t a science — but we’re getting closer
8. How other people use KISSmetrics
9. KISSmetrics is on it’s third iteration
10. Why KISSmetrics’ hasn’t launched
11. KISSmetrics tracks actions on a per-user basis
12. KISSmetrics lets you calculate customer LTV
13. Get more startup advice

Transcript

Music: Squarepusher

Trent Wolodko: Hey there everybody, my name’s Trent and I’m new to Venture Hacks. I’m going to be helping with producing content for the site so that the guys can concentrate on bringing you good advice faster and more furiously than ever. Today Nivi speaks once again with Hiten Shah from KISSmetrics.com. He spoke with Hiten recently about survey.io and today  he’ll be talking to him about optimizing your web apps using the tools over at KISSmetrics.  Right off the bat, these guys make a lot of references to Nivi’s interview recently with Sean Ellis about bringing your product to market as well as his former interview with Hiten regarding survey.io. So, you may  want to check out those interviews as well on the site. The guys have a lot of ground to cover as you can see, so without further adieu, I’ll throw it over to them…

Babak Nivi: So the second part of the interview with Sean is what activities you do to prepare for growth. Right?

Hiten Shah: Yeah.

After fit, prepare for growth

Nivi: One of the insights that I got from the interview was that you do a whole bunch of stuff to get to fit. That’s step one. Then step two is that you do a whole bunch of stuff to get ready to prepare for growth.

Hiten: Yeah.

Nivi: And Survey.io is great for the product/market fit stage, where you’re trying to get the fit. And you guys also have a product called KISSmetrics, which helps you prepare for growth, or to put it another way, to optimize.

Hiten: Yup, and that’s our main product.

Nivi: So, what does it do?

KISSmetrics helps you optimize your funnel

Hiten: First, it’s an analytics product, pure and simple, but the difference is that it’s meant to do one thing really well, and the thing it does really well is help you optimize a funnel – help you understand what your funnel looks like for whatever type of product you have.

Nivi: So, Sean has a few steps in the optimization stage, right?

Hiten: Absolutely.

KISSmetrics helps with all optimization steps

Nivi: Economics: put into place a business model and optimize it. For example, change your price or change the way that you charge, completely, whether it’s by the year or by the pound or however. So that’s the economic step. That requires optimization.

There is the positioning step – how you describe the product – which also requires optimization.

And then there is the step of essentially moving people through the funnel and getting them to the finish line. That requires optimization.

Hiten: Yup!

Nivi: How do I use KISSmetrics in any of those?

Hiten: You can use them in all of them, because when you have economics in place it means that you’re charging customers and that you need to understand things like how different price points affect your revenue. In essence, it’s a funnel. You’re still funneling people through a sign-up process, and you might be A/B testing two different things.

So we facilitate some level of A/B testing today, because you can pass in the data and visualize it, but in the future we’re going to start adding more components around A/B testing and optimization. Right now it’s a hard focus on helping people understand their funnel, but then that goes into both of those areas we were talking about.

So, in the economic stage it’s just about when you’re testing different pricing plans and things like that, and understanding if there are any differences, also, in different channels of customers depending on where they came from, and if your revenue is different for those based on the price points.

Then on the promise side you’re probably doing a lot of things around testing landing pages and things like that, so you’d build different funnels for different landing pages and different cohorts of users, and we completely facilitate that, as well.

Nivi: So you let me – and maybe you can just send me a couple screen shots that I can put in – compare two cohorts, how many people get to the finish line based on their channel, based on the positioning that they saw on the landing page, based on the final price or business model that they saw that we offered them, and also based on what Sean calls that “first experience.”

Hiten: Yeah, first user experience.

First user experience isn’t necessarily the paid user experience

Nivi: The first user experience, which is just kind of the funneling. I mean, it’s everything.

Hiten: Yeah, but it’s primarily how to optimize your on-boarding process of first-time customers so you can make that more efficient – efficient meaning highest conversion rates – and so that you can spend on marketing and you understand what the value is of each user.

Nivi: What does that mean to you, other than a conglomeration of the landing page and all the steps between that and putting the credit card in or signing up?

Hiten: In every product there are a number of steps people take to get gratified with the product. Gratification, usually in freemium products, happens during the free period, most likely, that instant gratification, and then they have another reason to go pay. So it’s all about understanding how people are moving through those stages of their usage of your product. So the idea is to track all the different events of gratification and optimize against that. And there are always steps in between before they’re gratified.

For the first gratification, they have to sign up. They probably have to fill out a form. They probably have to install something, or depending on what type of product it is there are a number of things they need to do before they feel satisfied that this is a must-have product. And once they hit that point, then you’re going to have an ask of asking them to upgrade, and making sure they hit a point where…. It’s really about you trying to understand what point they hit before they’re willing to give you money, and understanding the different things you can do around that to encourage them to give you money.

This all has to do with, if you have free plans, you’ve optimized your first user experience. Now you really need to understand what the prompts are and where you put them to get them to pay you money. So it’s all about prompt optimization.

An example is if you have an analytics product and you have a free plan, but you want to charge users, you probably need to put prompts in different places, as customers do different things with the product, to get them to pay, basically.

Nivi: In order for KISSmetrics to be useful to me, should I be done with the product/market fit stage. In other words, should I have 40% must-have users, or can I use KISSmetrics earlier, effectively?

KISSmetrics is most valuable after fit

Hiten: The short answer is that I think KISSmetrics is most valuable after product/market fit. But, this is one of those things where creating a startup and the steps to creating a startup aren’t defined yet. People look at them in all different ways.

I think both Sean Ellis and Eric Ries are making a lot of noise and movement, and there are a few other people like Dave McClure that are doing all of this, to get people to think of it more as a methodology, and there are different stages to a company and different things you should do at each stage. So if you follow one of those methodologies or one of those mindsets or theories, what you really want to do is wait until you have product/market fit before you optimize first user experience.

My own example is that we’re not optimizing our first user experience for KISSmetrics right now. We’re getting users in, but we’re not optimizing it because, really, that’s not the important part right now. The important part for us is getting enough users and building and iterating on the product to make it so that it’s something they can’t live without. And that’s not a quantitative thing; that’s not something where if I can measure how they go through a funnel, it’s going to necessarily help them. It’s more about things I need to add in the product, and it’s a little more of a fuzzy goal. Honestly, hitting product/market fit, in my opinion, is a pretty fuzzy goal.

Nivi: Tell me how Ash, who has written about this…What’s the name of his company? CloudFire? Is that right?

Hiten: Yeah.

Nivi: How is he using KISSmetrics actually in the product/market fit stage? Is that right?

How Cloudfire uses KISSmetrics during fit

Hiten: Yeah. What he’s actually doing is combining a few ideas. Dave McClure has something called AARRR!! that’s a framework that he came up with years ago. He’s also one of our advisors, for a good reason. And he spread a lot of the religion of metrics, so a lot of people say he’s the church of metrics.

But basically, what it does is break down your business metrics for a startup into 5 different things.
Acquisition, which is metrics around how you acquire customers, what channels are profitable or what channels are bringing in customers and how you’re bringing them in. So it’s more like market.

Nivi: So you define acquisition.

Hiten: Yeah.

Nivi: One definition might be: they signed up.

Hiten: Yes, exactly. They signed up, and where they signed up from.

What Ash has done is taken acquisition and made a funnel for it, and he calls it the acquisition funnel in KISSmetrics.

Then the next one is Activation, which is how to get them to have their first gratifying experience. They’ve activated with the product. They’ve used it. They’ve experienced it.

Nivi: And in his case it’s something like, they downloaded my software and uploaded some photos with it. Something like that.

Hiten: Exactly! Exactly. It’s like they activate it, which means they downloaded it and uploaded a photo. They had the first experience of using it.

And then the next one is Retention, which is that they came back or they used it multiple times or some factor of that. It depends on what your startup is. So for him it may be that they used it once, but then they used it a second time and that would be retention.

Nivi: In thirty days, say, they use it again.

Hiten: Yeah. So he’s got another funnel around that.

And then the next one is Referral. That one is more like, are they referring other users? And that’s a whole thing in itself; it’s ‘forward to a friend’ and ‘contact importer’ kind of stuff. There are all kinds of things around referral.

Nivi: Tweet about it.

Hiten: Yeah. That also gets into virality and all of that kind of stuff.

And then the last one is Revenue. So he’s got some revenue funnels around all of the prompts. I don’t know if this is exactly what he’s doing, but this is what I would imagine he would be doing. I don’t look into his account; I just talk to him and read his blog posts.

Nivi: That’s good.

Hiten: He’s probably got a funnel that’s focused around the prompts that he’s using to get people to pay. So he’s actually using it [as an] analytics tool, in general. People end up using it whatever way they want, and that’s one of our challenges, which is that we really need to figure out what type of use-cases we want to support.

Nivi: Question number 6 or 7.

[laughter]

Hiten: Yeah. You got it. So we’re not even there yet of giving out that survey, but we will be soon enough, hopefully.

Nivi: I think it will be interesting to see how his activation and retention numbers tie into the must-have segment.

Hiten: Agreed. Yeah.

Nivi: Probably mostly on the retention, if I’m thinking about it right now correctly, because for the person to even be a user that you would survey with Survey.io, you’d probably want them to already be activated.

Hiten: You might even want them to be a retained user, and hit the retention step.

Nivi: Yeah, you’d want them to be retained. Right.

Startups aren’t a science — but we’re getting closer

Hiten: So I think he’s actually using it in a very interesting way that we’re probably going to study a little bit deeper, because Dave’s an advisor and we buy into his theories. Actually, our first iterations of this product, which not many people saw, each of those was a view in our system, so we’re actually very intrigued by what he’s doing with our product. But again, I don’t think startups are a science yet, and I think we’re getting closer and closer to it, so it just really depends on what people subscribe to. All we can provide is a tool at the moment, and then hone in on these different use cases of it.

Nivi: Right. One thing that Sean actually mentioned in the interview is that he doesn’t see a lot of correlation between the recommendation figure and “success of the startup.”

Hiten: Right.

Nivi: Whereas he sees a fair amount of correlation between the success and the must-have figure. So that’s one thing to keep in mind for Ash, perhaps.

Hiten: Sure.

Nivi: And then, too, just in your case, like you were saying, with KISSmetrics, the way you’re bringing it to market is very feature oriented right now, and then you’ll figure out the positioning based on qualitative conversations with customers.

Hiten: You’ve got it. Right now it’s more about product and how to tweak the product. I hate to call it features because for us, actually in the market we’re in, every analytics product has hundreds of features. We actually want to provide the features that matter, so that’s kind of the difference.

It’s basically iterating on the product and making it so people like Ash love it. There are probably a number of things… He might already love it. He might be a very unique case where he’s been looking for something like this. But a lot of other people, I get on calls with these people or talk to them, and….

Nivi: So how do other people use it?

How other people use KISSmetrics

Hiten: How they’re using it is that there are several Facebook applications that are using it. They’re using it to optimize their own funnels and just optimize the conversion rates between each step, and things like that – normally what you would do with funnel optimization.

And then other folks, like SaaS businesses that are using it, are buying traffic from AdWords and they’re passing in the dollar value for their customers at one of the steps, and they’re just basically optimizing the funnel, but based on revenue because that’s what their main goal is and they’ve already got their revenue set up.

Nivi: So, visualize the funnel and visualize cohorts, and compare them.

Hiten: Yeah. And some of this stuff is still things we need to do better in our own product. We’re very early. We released this version on November 1st, so we’re not even a month-and-a-half into it.

Nivi: You guys have basically realigned the company around this product, is that right?

Hiten: We’ve realigned the company around this product through doing our own customer development this whole time – “this whole time” meaning this is our third iteration of the product and this is the one that we’re being a lot more public about.

Nivi: What were the first two?

Hiten: With our first one we had assumptions around doing metrics for Facebook applications and social network applications. And what we realized there was that we had a very constrained view, and it was based on Dave McClure’s AARRR!! stuff that we just talked about. What we realized about that was that people started asking all kinds of questions that our interface could not answer for them.

So we took what we learned there and we built a second product, which was very super-flexible event tracking, and we had no funnel analytics or anything like that, so this kind of stuff would help you with tracking engagement of users. If you were a game it would help you track game plays by gender and all of those kinds of things. And if you were a SaaS app, it was more about if Ash needed to measure what types of photos people were uploading or what kinds of things they were storing on his system, we would provide metrics around that kind of stuff.

Nivi: And what was that called?

Hiten: It was KISSmetrics. All of these are KISSmetrics, we just didn’t publicly release them.

Nivi: OK, got it!

Hiten: And so we used these other two products to kind of learn about the market and figure out what our target should be. And it was super-generic products, especially the second one. The first one was very targeted towards a market, then we kind of ran away from that and said, let’s get Facebook apps, but let’s also get SaaS applications; let’s get blogs using our product. And that was a very highly flexible product.

In some ways it would have been really fun to market that product and deal with it, but we were just heading down a path, in terms of sales and marketing and things like that, that we didn’t want to. As a business our DNA was more around building a really simple solution.

We have a previous analytics tool that the people in our company built called Crazy Egg, and it’s a simple tool, and we wanted to go back to that thinking and say, analytics is something where people log into analytics systems, and a lot of the time if they’re not expert users they’re very confused about what they’re looking at, and then they end up just using it as a trending tool just to see general trends.

And the kinds of features that we felt were most important were conversion goals, conversion rates and funnels. So with all this learning from our previous products we basically decided that we should hone in on that proposition, and instead of tacking funnels onto an analytics product, like the other one, we would just make that the product. And it was a big bet. I’m actually super excited about it and very happy that we went in this direction because I think we can really make an impact on the market and produce something really useful for customers.

Nivi: The feature is the product, here.

Hiten: The feature is the product, and that’s a trend. I mean, Twitter is just a feature, right?

Nivi: Right.

Why KISSmetrics’ hasn’t launched

Hiten: A lot of people have said, “Oh, you guys are creating a lot of buzz. We know your brand, but what the hell is your product, and when are you going to launch?” And for us it’s not about that. It’s not about launch; it’s not about things like that. It’s about building a product that people love and iterating until you get that.

Nivi: Yeah, and this is a good example of why you probably don’t want to launch until the business is working.

Hiten: Yeah, exactly. We don’t even have our economics down, so we’re not charging anyone right now, and everyone can come in and use the product. At some point we want them to help us figure out what the value is and what we can charge for it.

The way most analytics packages charge, people are penalized for being successful, and we want to see if we can change that a little bit. I don’t know what it means yet. I don’t know what it means for our pricing model, but I’d love to make it so that if you get more traffic we don’t necessarily charge you more. It sounds really fucked up and weird, but yeah. Excuse my language, but we want a better pricing model if we can find it, that’s more aligned with our customers. So there might be more features that they pay for, and it might be access to data or something that’s not exactly like: you get more paid views or you get more growth and we charge you more money, and it scales up like that. Even for us, that’s not necessarily the most scalable business model.

Nivi: Sean talked about using action tags with AdWords, in the interview, to track users on a per-user basis. We talked about Google Analytics, which doesn’t really track users on a per-user basis. What does KISSmetrics do differently here?

KISSmetrics tracks actions on a per-user basis

Hiten: The things we’re trying to solve are if someone comes to the site multiple times and then they sign up, let’s say on the third or fourth time, we’re allowing you to identify that user and we’re trying to attach all of those times that they came, previously, to that customer. What that means is that we’ll understand the first touch point they had with you and where it came from, and we’ll be able to attach that for the history of that user using your product. And that’s a much different view than most analytics have, which is either page-view tracking or event tracking. This is actually, honestly, people tracking, which means that we’re trying to actually track people.

Eric Ries has a great quote on this which is, “Metrics are people, too,” and we firmly believe that. If we can track things on an individual-user basis without getting into a lot of privacy concerns, because you’re already doing this in your data base, so you just have to have the proper terms-of-use for it, we can provide a much more valuable analysis on your user base. And as long as we’re not doing anything in aggregate there’s not any real privacy implication, while with Google Analytics I feel like it’s very hard for them to do that because they have a huge business that has nothing to do with analytics, in a lot of ways.

Nivi: Right. Does tracking people on a per-user basis let you calculate LTV, and to put it another way, cost of customer acquisition, and does Google Analytics not let you do that?

KISSmetrics lets you calculate customer LTV

Hiten: I’m not going to say that Google Analytics does not let you do that. We definitely think there’s a lot of value in the product, and we actually use Google Analytics in a lot of our products, and they do different things than we do. But legally, you cannot pass personally identifiable information into Google Analytics. So, if you’ve got 100 conversions in Google Analytics, you might have 110 in your data base simply because you’re not able to identify each individual user and they might lose some of the data.

And they’ve got, my guess is, 100 million sites using the product. That’s a high load. We’re probably never going to have 100 million sites using our product. There are a lot of issues that come around dealing with that many customers with a free product.

Nivi: Yeah. What I’m just trying to understand is what the requirements are to calculate cost of customer acquisition. Have you guys thought about that at all?

Hiten:  Yeah, absolutely. We want to make it as simple as you passing a dollar value every time an event occurs where someone’s paying. And then in our interface, it’s not like magic, but it’s magic. We can show you revenue by referrer, revenue by channel, revenue by plan type, and all of those kinds of things that typically you would have to do a lot of data base queries in your own data base, your user data base, to get all this data out.

And honestly, this is some of the pains we had with our previous products. So, for the first 6 months, we never tagged referrer for each customer that came in, so we couldn’t boil it down to what referrers were creating the most revenue for us. Once we started doing that it was enlightening. So KISSmetrics is actually based on a lot of the pains that we had with our previous SaaS product.

Nivi: So the dollar value that you pass to KISSmetrics will let you calculate the LTV, to some degree, of the customer, right?

Hiten: Yeah, that’s the goal.

Nivi: But not on the cost side. It doesn’t tell you how much it cost you to get the guy.

Hiten: Not today.

Nivi: Right.

Hiten: That’s a really good question, and that’s something that we do want to answer in the future. Obviously it depends on what type of customers we get and how they want to pass in the data. Is it automatic? Is it manual? Things like that. But it’s a problem that we’re thinking about a lot because then you can have a full picture in one dashboard. It’s all about that one report to rule them all, which doesn’t exist in analytics today. And if it does exist, it’s an internal report that’s built by your development team, and that’s kind of the problem we’re trying to solve.

Nivi: Cool! Thanks again for spending the time talking and also sponsoring the interview with Sean and making it free for everyone.

Hiten:  Absolutely! Our pleasure.

Get more startup advice

Trent: You’ve been listening to Hiten Shah, from KISSmetrics.com on how to optimize your web apps. And if you want to find out more, you can check out the KISSmetrics.com website. You can also follow Hiten on Twitter @hnshah, or check out his personal site at hitenshah.name.

Be sure to also have a listen to Nivi’s interview with Hiten regarding Survey.io, and the two-part interview with Sean Ellis on bringing your product to market, on the VentureHacks.com website.

Of course, if you’re into all other sorts of good advice for startups, head on over to Venture Hacks and just soak it all in.

Thanks again for stopping by. See you next time!

What you charge for your product is simultaneously one of the most complicated and most important things to get right. Not only does your pricing model keep you in business, it also signals your branding and positioning. And it’s harder to iterate on pricing than other elements of your business. Once you set a price, coming down is usually easier than going up.

Should I charge for my MVP?

Most people choose to defer the “pricing question” because they don’t think they (or the product) are ready. Something I hear a lot is that a minimum viable product is by definition (embarrassingly) minimal. How can you possibly charge for it?

A minimal product is not synonymous with a half-baked or buggy product. If you’ve followed a customer development process, your MVP should address the top 3 problems customers have identified as important and it should do it well. You can ensure that by dedicating 80% of your efforts to improving existing features versus cranking out new ones.

Steve Blank bakes price exploration right into the initial customer interviews. Price, like everything else, is built on a set of hypotheses that needs to be tested early. Steve suggests you ask potential customers if they’d use the service for free. This is to gauge if the product’s value proposition is compelling at all. You then ask if they’d use the service for $X/yr. How do you come up X? You can simply roll the dice and adjust along the way, or use Neil Davidson’s excellent guide to software pricing to start with a more educated guess. Once your MVP is built, Steve asks you to sell it to your early customers. There is no clearer customer validation than a sale.

Sean Ellis, on the other hand, argues that achieving initial user gratification (product/market fit) is the first thing that matters and suggests keeping price out of the equation so as not to create unnecessary friction:

“I think that it is easier to evolve toward product/market fit without a business model in place (users are free to try everything without worrying about price). As soon as you have enough users saying they would be very disappointed without your product, then it is critical to quickly implement a business model. And it will be much easier to map the business model to user perceived value.”

Both Steve and Sean advocate removing price from the equation — but at different points. Steve removes price during the customer discovery process but suggests you charge for your MVP. Sean removes price from the MVP and suggests you charge after product/market fit. I can see the merits of both approaches and wondered which was right for my product: CloudFire: Photo and Video Sharing for Busy Parents.

Why not use freemium?

On the surface, freemium seems like the best of both worlds: Get users to try your service without worrying about price, then up-sell them into the right premium plan later. However, many people make the mistake of giving away too much under the free plan, which leads to low or no conversions. It’s human nature — we all want to be liked.

More important, we don’t yet have enough information to know how to price or segment the feature set. I made this mistake with my first product, BoxCloud: an early visionary customer called me up and said, “I really like your product and want to pay for it but your pricing doesn’t require it.” After a few more iterations of segmenting the feature set, I decided to forgo the free plan and simply offered premium plans with a trial period. Sales went up and so did the quality of feedback, which I attribute to the difference between feedback from customers versus users.

(Hiten Shah shared a similar story with me around his experience with Crazy Egg. Even 37signals has greatly deemphasized their free plans to almost being fine print on their pricing pages.)

Lincoln Murphy just published a timely white paper on “The Reality of Freemium in SaaS” which covers many important aspects to weigh when considering Freemium, such as the concept of quid pro quo where even free users have to give something back. In services with high network effects, participation is enough. But most businesses don’t have high enough network effects and wrongly chase users versus customers. What I particularly liked in this paper is Lincoln’s recongition that “Freemium is a marketing tactic, not a business model.”

I strongly feel that, especially for SaaS products, starting with free and figuring out premium later (all too common) is backwards. If you know you are going to be charging for your product, start by validating if anyone will pay first. There is no better success metric and it leads to less waste in the long run. Focusing on the premium part of freemium first lets you really learn about your unique value proposition — the stuff that will get you paid. You can then come back and intelligently offer a free plan (if you still want to) with more intelligence and the right success metrics clearly defined. Even if you think you have a one-dimensional pricing plan like I did (e.g. number of projects), you’d be better served testing it with paying users because pricing experiments take a much bigger toll than other types of experiments.

Testing price in interviews

How did I put all this to test? The biggest mind shift in following a lean startup process is going from thinking you know something to testing everything you think you know.

So I followed Steve Blank’s advice and built some pricing questions into my initial face-to-face customer interviews. Because CloudFire is a re-segmented product in an existing market, potential customers referred to competitor pricing. This had to be balanced against the perceived value of our unique value proposition – saving time with faster and easier sharing of lots of photos and videos. Through these interviews I determined that, like their sharing needs, my potential customers valued simple hassle-free pricing and $49/year for unlimited photo and video sharing was a fair price they were willing to pay. That is what I charged them once my MVP was ready.

Testing price on the web

I wanted to run the same set of pricing tests with web visitors that I did during my interviews. Short of split testing a free and paid version of the MVP, which is technically illegal (update) and unfair to paying customers, I decided to split-test 3 different products with 3 different prices:

1. $49/yr for unlimited photo and video sharing
2. $24/yr for unlimited photo sharing
3. FREE for 500 photos

All plans have a 14-day free trial with the exception of the free plan which is free forever. Here are the variations I tested:

Original: Single unlimited plan

This is the simple option I discovered during customer discovery interviews. It served as the control.

Variation 2: Multiple plans

I segmented the product into 2 offerings: unlimited photos+video and unlimited photos only. I wanted to test price sensitivity and gauge interest in video sharing. Not many people I interviewed were currently taking lots of videos but they all wanted to be taking more.

Variation 3: Freemium

This has the 2 plans from above along with a limited free plan. Yes, this is a freemium plan. I wanted to measure if a limited free plan would disproportionately drive the right type of traffic (busy parents in my case).

Variation 4: No Price During Introductory Period

I added a fourth variation to test Sean Ellis’ advice on removing price till product/market fit, but I tested this differently. I was not comfortable offering the full product for a price and for free at the same time. So rather than including this page with my A/B tests, I instead tested it with new parents I interviewed.

The Results

First Place: The original single plan — second place in conversions and best overall performer. Surprisingly, the original page was the best overall performer.

Second Place: Variation 3: Freemium – most conversions but second place overall. Not surprisingly, the freemium variation drove the most conversions but only outperformed the original by 12% and had the lowest retention. Referral stats combined with random polling/emailing revealed a majority of the users that signed up were just curious (and not parents).

Third Place: Variation 2: Multiple plans – least conversions and worst overall performer. People reacted least favorably to the two paid plans.

Non-starter: Variation 4: No price during introductory period. Parents I interviewed did not understand the introductory period without explanation and were reluctant to try the service without knowing how much the service was going to cost. Probing further, they weren’t willing to invest the time building up web galleries and inviting others only to find that the service might be priced out of their expectation.

What I learned

It does pay to align pricing with your overall positioning. Our unique value proposition is built around being “hassle-free and simple” and people seemed to expect that in the pricing model as well. A lot of our existing customers were already paying for their existing sharing service so the leap from free to paid was not a big one. While Sean suggests removing price before fit for consumer facing products, he suggests always charging for enterprise customers to gain their commitment. This is another case where pricing needs to be explicit. Using Cindy Alvarez’s model, our customers appear to be Time-Poor, Cash-Rich. Offering no-hassle free trials was sufficient to remove the commitment risk. Money back guarantees might be another way to further lower this risk.

The biggest lesson learned, though, is how accurate my initial customer interview findings were, compared to all the hypotheses that followed. Pricing is more art than science and your mileage will vary, but whenever possible get out of the building, talk to a customer, and consider testing price sooner rather than later.

What do you think? Why do you think these variations finished the way they did? What other variations would you like to see us try? How else do you think we could increase conversions? I’m looking forward to discussing your responses in the comments.

That’s what I’ve been hearing since we published Part 1 of our rare interview with Sean Ellis.

Here’s part 2.

In Part 1, Sean discussed what you do before product/market fit: how to get there, how to measure it, and how to survey your users so you can improve fit.

In Part 2, he explains what you do after fit: optimizing your positioning, implementing a business model, and optimizing your funnel — all so you’re prepared to acquire users quickly and profitably.

If you don’t know Sean from his blog or tweets, he lead marketing from launch to IPO filing at LogMeIn and Uproar. His firm, 12in6, then worked with Xobni (Khosla), Dropbox (Sequoia), Eventbrite (Sequoia), Grockit (Benchmark)… the list goes on. 12in6 “helps startups unlock their full growth potential by focusing on the core value perceived by their most passionate users.”

This is the first time Sean has done an interview on the record. I’m really psyched he’s making his insights public — this interview is a must-listen.

SlideShare: How to bring a product to market, Part 2
Audio: Interview with chapters (for iPod, iPhone, iTunes)
Audio: Interview without chapters (MP3, works anywhere)
Transcript with highlights: Below

This inteview is free — thanks to KISSmetrics

We’re bringing this interview to you free, thanks to our sponsor KISSmetrics.

Sean is an advisor at KISSmetrics and we interviewed their CEO, Hiten Shah, in How to measure product/market fit with survey.io. KISSmetrics built survey.io with Sean — now they’re collaborating on KISSMetrics, a new tool for funnel optimization that we’ll discuss in an upcoming interview with Hiten Shah.

Prerequisites

You’ll get more out of this interview if you also read:

1. Part 1 of our interview with Sean.
2. The startup pyramid.
3. We use several phrases interchangeably in the interview: Growth = scaling = acquiring customers with a known ROI. Preparing for growth = after product/market fit = optimizing promise + implementing economics + optimizing the funnel.

Outline

Here’s an outline and transcript of Part 2.

1. What comes after fit?
2. Prepare for growth
3. a) Put the metrics in place
4. b) Optimize the funnel
5. c) Optimize the messaging
6. Prepare for growth as quickly as possible
7. Remove bottlenecks to preparing for growth
8. Preparing for growth is not a low-burn period
9. Preparing for growth doesn’t require growth
10. Use a business model to grow quickly
11. Grow and create new channels
12. Nail the first user experience while preparing for growth
13. Leave no room for the competition
14. Marketplaces are an exception to this model
15. Why Sean decided to focus on startup marketing
16. Just scratching the surface

Transcript

Music: The Equator by Tortoise

What comes after fit?

Nivi: Right. One thing you talk about is, once you have the product/market fit, trying to get through the next few steps as quickly as possible. So it would be great if you could talk about that, and also, for people that don’t know what the pyramid looks like, what are the next few steps?

Sean: Sure. As I said, if you don’t have product/market fit, you want to be obsessively focused on getting there, however you’re defining product/market fit. I think the easiest kind of definition to work toward is trying to get 40% of your users to say they’d be very disappointed without it.

And during that period you’re trying to stay very low burn and very conservative on all of your expenses within the business. And once you find 40% of your users that say they’d be very disappointed without the product, then you’re in a position that you have a business that can grow now. So then the question is: do you try to grow the business right then or are there some things that you can still do that are going to make it even easier to grow when you’re focused on that? And what I’ve found in working with so many companies at this [post-fit] stage is that you’re definitely better off waiting a little time until you try to grow the business.

Nivi: And what does it mean to grow the business? You’re talking about spending money to acquire users, essentially.

Sean: Acquiring users, for a lot of businesses, means starting to spend money, but it might also mean really trying to crank up the virality of the business, or it might be SEO that doesn’t require a lot of spending.

Nivi: But it’s time and money that you require for people… [interrupted]

Sean: Yeah, and focus. The time piece, you could say, has time or focus. But it’s essentially saying…

Prepare for growth

Sean: Is now the time to orient — especially the CEO and the marketing group and maybe a sales group — is now the time to build up that group and just say, let’s step on the accelerator and do everything that we can to get very aggressive about growing this business? And I would say, not quite.

The reason being, one, up until this point you shouldn’t have really focused on really trying to get the metrics into your business, so one of the ways that you can effectively grow a business today is, you can experiment in lots of different areas and figure out, in which areas do I invest a dollar and get that dollar paid back the most quickly or make a really strong return on the investment? But you can’t do that if you don’t have the measurement systems in place. So one of the first steps that you want to do is specify what metrics you really need to be tracking in order to be able to grow this business. And you want to then work to implement those.

So, to some degree, Google Analytics can help you, but the problem with Google Analytics is that you don’t really have tracking on a very user-specific level. So maybe you can track with Google Analytics or with Website Optimizer, even with Google AdWords with action tags, you can track a first transaction through dollars spent, but what you can’t track is lifetime value. And ultimately, expected lifetime value is what you should be basing your customer acquisition investments on.

Nivi: With the per-user tracking, is that what KISSmetrics is for?

Sean: That’s what KISSmetrics is working on. For other companies that I’ve worked with, it’s basically been database driven reports where we’re cookie-ing users when they’re coming in to track them back to the source, and we’re recording that information in the user’s permanent record in the database, and that way any revenue that’s generated from that user, we’re able to tie back to the money that was spent on that referring origin, coming in.

The reason why I started working closely with KISSmetrics is that all of my projects were six months, and that type of a system that I just described tends to be fairly complicated, especially as you start to track where you’re losing users along the way, and some of the other pieces. It was taking a big chunk of the time that we had together to implement that, and there were a lot of bottlenecks around it, so I was trying to find someone who could build that type of a solution as an off-the-shelf type solution and KISSmetrics was on their way to doing something like that. So I’ve been advising them for a couple of years to try to get the full vision out there of something that can really give the metrics that any business needs to be able to drive and manage growth.

Optimize the funnel

Sean: So metrics is just one part of it. Obviously, as I started to touch on right there a second ago, most companies initially have a very inefficient customer-acquisition process. So it’s not just how effectively can you spend money externally, but how can you convert people once they get to your website to having a gratifying first experience, and then ultimately buying your product? And metrics are a good way of determining experiments that you run along the way — how can I get more and more people to actually convert?

That can be through landing-page testing. You can be really effective on that front, or all the way down to just testing purchase prompts. There are all different areas, so you want to have a pretty sophisticated measurement system in place to do that.

I’ve worked with businesses where, at the start of that process to the end of that process, in a matter of just a couple of months, we’ve seen five times as many people purchasing just by more efficiently on-ramping them into the product and converting them into being paying customers, which means that when we then focused on trying to grow the business and buy growth, we could spend five times as much money to get someone to the website at the same return on investment, after that process.

And when you can spend five times as much money to get someone to the website, there are a lot more viable marketing channels that are open to you — a lot of times channels that would not have been effective at all before. If you’re putting a dollar in and you’re only getting 50 cents back, after you’ve gone through this process and you try that channel again and now you’re putting a dollar in and you’re getting $2.50 back, you’re going to put as many dollars as possible into that channel, and you would have cut it previously.

Nivi: And the expected LTV of the user is equal to your allowable acquisition cost per user. Is that right?

Sean: That would be your allowable acquisition cost for a break even. It kind of depends on what your objectives are. If you’re trying to do market share you might even be buying users at a loss initially. If you’re trying to generate a profit, you probably want to build in some profit on that.

Optimize the messaging

Nivi: Right. So, we put the testing in place. What other steps are we going to do before we grow?

Sean: So messaging. You want to make sure that you’ve got good messaging. So part of that was to get the product/market fit. You might be doing some positioning and messaging. But this is where you want to do a lot of landing page optimization and fine tuning just to see, what’s the best way to convert users coming in?

Nivi: In terms of positioning.

Sean: Ultimately in terms of converting them, positioning is part of that. To me, positioning is sort of the qualitative side and then testing and optimization is the quantitative way of getting to the best result. Hopefully, where you end up is somewhat consistent with what your positioning usage led you.

Prepare for growth as quickly as possible

Nivi: And what are your thoughts on just going through this whole process as quickly as possible?

Sean: That’s a good question. The mistake that I’ve seen a lot of companies make, particularly those that struggled to get to product/market fit, is that the whole mentality while you’re trying to get to product/market fit is to be super conservative and not spend very much money, so a lot of companies kind of [say]: OK, we got there. Now let’s prepare to grow.

And they, again, are really conservative and they’re taking six months to…. A prime example would be: I need to spend $100 a day or $200 a day to know that this is the best-performing landing page, or home page even, or I need to send 1,000 people or 10,000 people through there, and most people will space it out over two weeks. They’ll say they don’t want to spend more than $100 a day while we figure that out, so we’ll space it out over two weeks.

In reality, if you have some place where you can spend the money — let’s say you’re doing that through a Google campaign and you’re not getting the cap on that, but you’re saying, I just don’t want to spend more than $100 a day, you’re essentially going to spend exactly the same amount of money to get to the answer if that’s a good page or not. One way it’s going to take you two weeks; one way is going to take you one day. If you consider that time is money as well, then you’re closer to being able to accelerate the business after one day than you would have been otherwise and it costs you the same. But it’s just really hard I think, a lot of times, for people to go through that mentality.

Remove bottlenecks to preparing for growth

Sean: Another example would be if you are tight on graphic design resources internally, if you test 50 landing pages or 100 landing pages or 1,000 landing pages, you’re going to be so much more efficient at being able to drive growth in your business that it just doesn’t make sense, if your big bottleneck is on graphic design, then you want to bring in two or three people. Pay them a little bit more. It’s a temporary spike in costs to get much faster to the point where you’ve got a lot more landing pages in there.

So you combine those two things. We’re going to spend more to get people through, and we’re going to be able to test five landing pages at a time rather than just two. So being able to do all of those things… The goal should really be….

Preparing for growth is not a low-burn period

Sean: You have two low-burn periods in the business. The first is pre-product/market fit. You’re trying to spend the least amount possible.

And then the second low-burn period is once you accelerate the business you’re working within the parameters of your allowable acquisition costs, so you can actually go to being cash-flow positive pretty quickly at that point.

One of the companies that I worked with, once we got through this period of — it took us about four months to transition to being able to really accelerate the business and we got it to the point where we could scale marketing to close to a million dollars a month with a fast payback on those marketing dollars.

And at that point we had to raise some more money to be able to fund those campaigns. We raised the money on some very proven metrics and went cash-flow positive the month after we raised the money. So it shows that if we had taken 16 months to go through and really optimize that experience, it probably would have taken a lot longer to get the cash flow positive, but we were very focused and aggressive on it through that period of time and could really accelerate that business in a very cash-efficient way once we got through that period.

Preparing for growth doesn’t require growth

Sean: The mistake that most people make is they’re trying to manage growth and optimization at the same time. For this period of time, growth isn’t important. That’s the other thing that’s really hard for people to consider. That is not the objective — efficiency is the objective.

So for the first part of the pyramid, the base of the pyramid, you’re just trying to create a product that’s good enough, that people want, so growth is definitely not important.

A lot of companies still pat themselves on the back if they grow faster than expected through that period. You shouldn’t, because what happens, especially if you go to your VCs and you pat yourself on the back publicly to them, then you’re essentially saying growth is important to us, and you’re setting expectations that they should want more of it. So basically, say: Hey, we grew. That’s not what we were trying to do yet, but it’s positive on the hope for the business, but before we really focus on growth we want to get our product/market fit number to this.

Now, in this case, basically we want to be able to grow as efficiently as possible when we’re focused on it. And so what we want to do is build in efficiency in the business and the customer acquisition monetization.

Use a business model to grow quickly

Sean: So this is the time when you put in the business model as well, because you can’t spend real aggressively if you aren’t confident that you’re going to get that return on the investment.

A lot of companies make the mistake of… at least I believe it’s a mistake. This is probably one of the most debatable things that I say; a lot of companies believe they can grow faster without a business model. To me, having seen companies that can arbitrage dollars to grow and do it really effectively, you can just accelerate the business so much with that, why would you not spend money to grow? But you can’t do that if you don’t have a business model in place. So that’s why I tend to want to put a business model in place at that point.

The mistake of trying to put a business model in earlier, before you have product/market fit, is that you don’t know if you’re charging for the right things in the right way. You may be charging for things that are totally irrelevant for people and giving away stuff that is really important to people. So that’s one of the big reasons I like to delay the business model until, one, I have product/market fit, and two, I know why I have product/market fit — I know why people love the product.

Grow and create new channels

Sean: Now I’m in a position to put the business model in place, work on all the efficiency on the customer acquisition side, and do everything so that when it comes time to focus on growth, the marketing group or the CEO or whoever is going to be managing that process can be completely focused on driving growth — they’re experimenting with lots of channels and finding those that work and killing those that don’t, and basically just as quickly as they can, adding as many new customer acquisition sources to the mix.

And you shouldn’t focus on that until you know how to efficiently convert people and until you have a product that people really need. So that’s why this pyramid just gives you the idea that until you have really achieved that level, and if you’re trying to do multiple things at the same time, you’re not going to give things the necessary focus to actually accomplish them.

And your goal is ultimately to be growing a product that has very efficient conversion, that you’ve got all of the metrics in place to really manage that growth, you’ve got a product that people love, and at that point you’re starting to become a regular company. You’re organized into product teams and marketing and sales and finance and all of those things that are needed to create an efficient, high-growth company that can manage that growth and not implode under that growth.

That’s really why in that center period of time, it’s exciting. What should be driving you, day in and day out, is that we’re close. We’re right to the point where we’re almost going to accelerate the business.

Nail the first user experience while preparing for growth

Sean: So, one other piece that I think fits into this is that a lot of times it requires a lot of experimenting with how you introduce people to your product. What do you show them first? What is that whole first user experience?

The mistake a lot of companies make is that they try to get that first user experience right before they have product/market fit, so those companies are essentially pulling limited resources off of their core product and saying: we’re going to focus those resources on getting that first user experience right. Then that means that they’re not really dealing with the problem. They can maybe get twice as many people in to experience the product, but it still sucks. So they’re not really dealing with the issue.

But now everybody who comes in, or a big chunk of people who come in, are saying that they’d be very disappointed without this product — there’s a lot of love for that product. At that point, basically hitting the pause button on core-product development for two months isn’t even going to hurt the business because people already love it, and taking all of those resources that would have been focused on honing that core-product development, and refocusing those on: How do you introduce people to that? What does that first user experience look like?

The mistake that a lot of companies make is that they’ll take one person off to do that, and it’s going to take them a lot longer to do the development to get that right. But now you’ve got that core-product experience right, taking the majority of your development resources to get through it in a few weeks is going to get you there faster and that is going to put huge dividends on your business.

Nivi: You’ll become master of your own destiny much more quickly. Right?

Sean: Yeah. The way that I’ve been able to get development teams excited about that is, one, I give them the context of the whole big picture, and then I essentially say this is not something that we’re going to ask you to keep coming back and refining the first user experience. Once we’ve really got it right, backed up by numbers, backed up by saying twice as many or three times as many or ten times as many people are getting in and experiencing that core-product experience, then you can go back into just working on that awesome core-product experience that you’ve created, and evolving that and just continuing to create a great product experience.

But otherwise, we’re going to have to keep asking you for help for the next six months, until we get that first user experience right. It’s just much better for everybody, for you guys, to put the pause button on that, help us get this right, and then we’re going to be able to grow the business and it’s going to be a lot more exciting.

Leave no room for the competition

Nivi: And once you’re growing the business, what do you think about growing quickly to eliminate slack in the marketplace and basically leave no room for the competition?

Sean: Once you have product/market fit, and once others see you having traction, you’re going to attract a lot of copycats into the business. And they should be trying to come in because, like I said, the product/market fit is kind of the hardest thing to get to, so once somebody sees that you have a lot of passionate users who really love it, it’s the easiest way to create a moderately successful business — to knock that off.

So you have a short-term advantage, which is you’re able to respond to all the real product feedback that you’re getting from users who are coming in and experiencing that, so you can start to evolve your product better.

So partly what you want to do is just buy every related keyword, maxed out, and be very efficient on those keywords so that you can spend a lot more money on them acquiring users. So, the next guy can’t even consider spending keywords, so you’ve killed that channel for them. You just want to basically make it so that by the time someone can respond, it’s going to take them several months to be able to come out with something that’s close to your product. By the time they come in there, there’s just no market left for them to play in — you’re really redlining that market.

It’s interesting. In companies that I’ve been with through the growth period, the products that come easiest are the ones that often get the least focus because, oh, we’re hitting the targets on those numbers. We have to work harder on these other numbers. But it’s always the products where the growth comes easiest where the competition comes in because we didn’t take the slack out of the market in those products, and then suddenly competition that we didn’t expect, and then it’s not so easy to grow any more.

Marketplaces are an exception to this model

Sean: I think all of the things that I’ve talked through are a snapshot of what I think is best right now. There are so many exceptions in the different types of businesses that are out there. For example, marketplaces do need to focus on growth a lot earlier. Their business is a function of the people that are on there and how they’re using the product, and the experience changes for everybody else with more people on the product. eBay would be a good example of that.

I think each of these things are some of the guidance that I wish I had in the startups that I was in, but don’t just take it exactly as-is and think you can just plug it right on your business and that it will work. You need to sort of interpret what this means for your business.

Nivi: Do you have any experience applying any of this theory to marketplaces at all?

Sean: Yeah, Y Corp. is a company that I’ve worked with that is a marketplace that I’ve definitely learned a lot and learned that it’s different than a lot of the companies, like Dropbox or Xobni, where it’s a more specific product that, with a million users or fifty users, there would be experience where the product doesn’t change that much, where for Y Corp. the experience changed a lot.

Nivi: So how do you still take a thoughtful approach when you’re building a marketplace?

Sean: It’s just a lot more complicated. I think, for me, one of the things that’s kind of come out, in my head, on marketplaces is that you probably need deeper pockets for it. You don’t have the luxury of focusing quite as laser focused on each of the steps in the pyramid that I talked about — that you actually do need to think about growth earlier on, and that even when you’re then focused on growth you need to make sure that you’re constantly finding out if the love is moving, if it’s on something else. It’s just a lot more dynamic to try. It’s fun. It’s interesting, but it’s definitely harder.

And I’ve had enough conversations with other people from dating sites to anything that’s got the experience based on the mix of people that are there, that it’s really something that affects any businesses like those. So that’s one piece that, don’t just feel like you can plug this in directly. There are probably exceptions for a lot of these things.

Why Sean decided to focus on startup marketing

Nivi: Why did you decide to focus on startup marketing, and why is it important to you and how did you come to that conclusion?

Sean: It actually happened when I was coming off of LogMeIn and figuring out what I was going to do after LogMeIn. I knew that the company had gotten big enough that it was not really the size that I wanted to be working in anymore, and that I wanted to go back to early stage.

So it was more just thinking, OK, I can jump into another startup and hopefully I can be fortunate enough to get into a startup where it has the success. But just looking at realistic startup success rates, LogMeIn was the second in a row of two startups that I had run marketing from launch through a NASDAQ IPO filing, and I just knew, realistically, that there was not a good chance that I was going to be able to do that with a third one.

And so partly what I was looking at was just in general…. I kind of went through the thought process of: did I just get completely lucky, which certainly had something to do with it, or were there some things that we did that actually led to that success? And when I really started thinking through both of the companies, I came to the conclusion that the hardest part and the part that really mattered in building a successful company, was what we did in the first year, and really just figuring out everything about the business: Who really needs this product? Why do they need it? How are we going to charge for it? How are we going to acquire new users?

All of these things that needed to be figured out, once we had them figured out then it was much more about, OK, now how do we get more users? But a lot of the moving parts had stabilized at that point and we could really focus on just getting more new users. And so I thought, if that’s really the most important part, I thought that it was also the most challenging part, and so future success that I have is really going to be based on how well I can do that first part.

And then I realized that in 10 years, across these two startups, I’d only spent one or two years in that really challenging stage, so if I really wanted to get good at that I was going to need to figure out how to work in that stage a lot.

And the problem with startups is that every person working in a startup full-time, basically has an option vesting period that’s usually about four years on those initial options. So you work really hard in the beginning and you get some level of success and then it takes another four years to be compensated for that success, so there’s no way you’re going to create all this success and then leave the company right away.

So for me it was more about: how can I work really hard in that stage and actually be compensated for it? The only thing that I really figured would be the best way to structure it would be as an interim VP marketing role, and I was introduced, from one of the investors in LogMeIn, to a company that was in his portfolio called Xobni. They were still in private beta, and he said just talk to the guys. So we had a conversation that ended up extending for a couple of hours and we really bonded over some things.

And I got excited about the product and they got excited about potentially working with me and tried to bring me in long-term, full-time, and that was not something I wanted to jump right into. I was really passionate about figuring out this early stage and doing cycles there. So we decided that it made sense to pursue an interim VP marketing role. So I basically went there and thought, I’ll do interim VP marketing role after interim VP marketing role, for the next several years — essentially negotiate my exit before I came in so that I’d be able to keep a reasonable amount of options. That was the initial plan, and the whole idea was just to get the experience, up front, and practice, up front, to get good at that and to really document what’s important and what’s not important.

I think the challenge that I figured out was that, really, on the marketing side, there’s a lot of bottlenecks to get a lot of the things done, like optimization and trying to get resources to do a lot of those things. Everybody’s scrambling to get the overall product out.

So I knew a lot of what needed to be done, but there’s just a lot of waiting around until you have the tools that you need to do that. So where I’m looking at everyone around me working 14 hours a day, and wanting to put that time in myself, there just wasn’t that much to do. So I almost felt guilty that I wasn’t putting in the same amount of effort that everybody else was, so I still spent the time in the office, but I just found myself doing a lot of things that maybe weren’t that important.

That was why, after Xobni, I decided that I was going to do a more leveraged model where I could work with a couple of companies where they actually had somebody inside to manage the execution of things, but that I would actually be more on the outside, being able to not spend more time in the company than I needed to spend. So that’s why I did Dropbox and Eventbrite in that sort of way, where I was balancing two companies at once, and that worked pretty well.

Just scratching the surface

Sean: I feel like I’m still just scratching the surface on what can be learned in early-stage startups. There are not a lot of people that have deep enough experience and frequent enough experience in that really critical up-front stage to get good at it. It’s been fun to try and figure it out, but there’s a big variety of companies. There are certain things that I think are applicable to all businesses.

One of the most important discoveries in the last couple of years, since I’ve been focused on this stage, is figuring out this very-disappointed number and just understanding that trying to grow a business that doesn’t have users that would be disappointed without it, or don’t consider the product a must-have, is going to be a difficult and frustrating experience for anybody, including me. It’s humbling. I can’t do it.

I can’t do it very well, and when I see companies doing it I have a lot of admiration for them to be able to grow a business that is weak on that [fit] number. Those are the hard-core marketers that are kind of too good for their own good, because they’ve basically allowed the business not to focus on what the core issue is that’s holding them back. But those people would really benefit from taking a step back and tightening up some of these things.

I’m doing an experiment with Webs.com, for example, that is a later-stage company. They’ve been around for a long time. They’re a top-100 website and have a ton of user passion and a lot of the things that I’ve done to help prepare companies to grow are things that Webs hadn’t done, so we’re working together to see if we can retroactively put some of these things in place to accelerate their business.

And for me it’s all about learning and just experimenting in companies to find out what’s unique to startups and what’s unique to later-stage companies and what applies to everything.

Nivi: Great! That’s great. I just want to say thanks very much, Sean, for talking to us.

“Billy is perhaps best known for his Oxi Clean commercials, where one softball size ball of detergent, chucked into your washing machine and forgotten, will do 25 loads of laundry. Simple messages, great customer testimonials and products that really worked were Billy’s forte. He rejected perhaps 99% of the hundreds of inventors/companies that approached him, choosing only those products that met his three key metrics. And he and his inventors made millions in the process…

“Address clear customer needs: Billy wanted products that anyone could use, not just a select market segment–who does not have dirty clothes (Oxi Clean)? Needs a knife to cook (Samurai Shark)? Want shiny clean floors (Orange Glo)? or, my favorite, who has ever wished they could permanently cement their mother (or father)-in-law to a chair (mighty putty)?

“Have believable customer testimonials: Billy did not use actors, but recruited users who were true believers. Who could not like the housewives who had orgasms over their newly waxed floors?”

(Italics mark my emphasis.)

That’s a great set of requirements for picking startups: simple messages, great customer testimonials, and products that really work — that anyone can use. (You can also take the opposite approach on the “products that anyone can use” requirement.)

Bonus: Also see David’s post on Hiring the Right Sales Guy:

“So what’s the best type of sales guy for a start up? To be honest, none of the above. The founders are the best sales guys, supported by the correct marketing and sales support technology, processes and people. They know their solutions and vision best and should be the one who lead the charge in the marketplace.

“Instead of hiring a very expensive super sales guy, for example, use your board members to get introductions at target customers. That’s a prime role for board members and should be a key criteria in choosing them. Want C-Level introductions? Try cold calling at 8am in the morning when the C-levels get in early to check their emails. Need a group of specialists to sell the solution?  Assemble the team from current employees to make the pitch. Everyone should be a sales person in a start up.”

This is consistent with Steve Blank’s advice that founders should be on the customer development team and, if necessary, complemented by a ‘sales closer’.

“In the many thousands of articles advising entrepreneurs on what they have to focus on to build successful startups, much has been written about three key factors: team, product and market, with particular focus on the importance of product/market fit. Failure to get product/market fit right is very likely the number 1 cause of startup failure. However in all these articles, I have not seen any discussion about what I believe is the second biggest cause of startup failure: the cost of acquiring customers turns out to be higher than expected, and exceeds the ability to monetize those customers… [emphasis added]

“A quick look around all the B2C startups shows that, although viral growth is often hoped for, in reality it is extremely rare. When it does happen, the associated businesses are usually extremely attractive, provided they have a way to monetize their customers. (For more on the topic of Viral Growth, refer to my blog post on that topic here.)

“Far more common is a need to acquire customers through a series of steps like SEO, SEM, PR, Social Marketing, direct sales, channel sales, etc. that will cost the company significant amounts of money. What shocks and surprises many first time entrepreneurs is just how high the numbers are for CAC [Cost to Acquire Customers] using these kinds of techniques.”

Read the rest of David’s excellent post: Startup Killer: the Cost of Customer Acquisition. But don’t worry about it too much until you’ve built a product people want. It’s hard to know what the CAC is before you know what the product is. And check out the rest of David’s new blog — his post on Viral Cycle Time is great.

The comments to our interview with Sean Ellis turned into an awesome Q&A — this post is a roundup.

(Feel free to keep asking questions here or there and I’ll try to get answers from Sean.)

Is my product a nice-to-have or a must-have?

Jae Chung wonders whether his product is a nice-to-have despite the positive press:

“I spent the past 24 hours poring over each of the points [in the interview]. We also formed about 8 months ago and the site is currently undergoing beta testing and has received positive feedback from many of our users and the press. However, my gut tells me we are in the “nice to have” category, and could never quite put our finger on what it was that users found appealing. We’ll definitely be implementing your survey to find out where the “love” is!”

The survey he’s talking about is survey.io.

Should I charge users before fit?

Sean Ellis:

“I think that it is easier to evolve toward product/market fit without a business model in place (users are free to try everything without worrying about price). As soon as you have enough users saying they would be very disappointed without your product, then it is critical to quickly implement a business model. And it will be much easier to map the business model to user perceived value.”

Michael Harry Scepaniak:

“…freeing yourself (pun intended) of paying customers early on would seem to allow you to make more radical moves (pivots), since you don’t have to worry about angering anyone that has given you money and expects you to deliver on their expectations in return.”

Instead of charging users for a part of the product they don’t even want, first find the part they love, and then figure out how to get users to pay for it. Entrepreneurs who advise you to charge from day one probably had fit early on in their startups.

How do I tell users that I’m going to charge someday?

Eric Santos:

“Do you communicate to the users that the product will have a price someday?”

Sean Ellis:

“I would communicate that “it is free during Beta” or if “beta” is too techie, then free during the introductory period. If you plan to have a free version, you can also let people know that.”

Should I pay users to send feedback?

What if my customers aren’t filling in surveys?

Vincent Chan:

“From my experience, many SMB users don’t like to fill in a survey for an unknown startup. Should I take that as a bad sign? In other words, is the survey response rate an equally important metric as the “must-have metric”?”

Sean Ellis:

“Yes, I’ve found survey response rate directly correlated to the percentage of users that consider the product a “must have.” For “must have” SMB products I often see the response rate over 10%.”

How do I find the love in a hardware company?

Samuel Bouchard:

“Sean, how can this “find the love approach” apply to hardware companies? What needs to be adapted to the method when you sell a product that is worth several $k’s?”

Sean:

“Samuel, I haven’t worked on a hardware product, so I’d just be guessing… Given the cost of getting a hardware product to market, I’d spend a lot more time up front on “where’s the need?” Steve Blank’s book Four Steps to the Epiphany gives great guidance on this.”

If you can build a product in a day, show customers the product. If the product is going to take weeks, show customers a PowerPoint instead.

Play us out

Ryan Nile knows what it means:

“This basically describes what we need to do after the MVP is up.”

Thanks to KISSmetrics for supporting our interview with Sean Ellis. If you want an intro to KISSmetrics, send me an email. I’ll put you in touch if there’s a fit. Thanks. – Nivi

Hiten Shah from KISSmetrics recently sat down with me to explain how to use their product, survey.io, to measure product/market fit and find the “best grass” in your product. You may know Hiten from his Crazy Egg days.

survey.io is a free survey tool that helps you implement some of Sean Ellis’ techniques to get to fit. KISSmetrics actually built survey.io with Sean. Unfortunately, there hasn’t been great documentation for survey.io besides Sean’s launch post. Until now.

SlideShare: How to measure product/market fit
Audio: Interview with chapters (for iPod, iPhone, iTunes)
Audio: Interview without chapters (MP3, works anywhere)
Transcript with highlights: Below

Prerequisites

You’ll get more out of this interview if you also read:

1. An example survey from survey.io.
2. Our interview with Sean Ellis.

Outline

Here’s an outline and transcript of the interview. The interview and transcript are about 19 minutes long so we’ve highlighted some of the juicy bits to get you started.

1. survey.io: Before product/market fit
2. survey.io measures fit
3. How did you discover the product?
4. How would you feel if you could no longer use the product?
5. What would you likely use as an alternative if product were no longer available?
6. What is the primary benefit that you have received from the product?
7. survey.io is open-ended
8. The flock will always find the best grass
9. You don’t need survey.io to find the best grass
10. Have you recommended the product to anyone?
11. What type of person do you think would benefit most from the product?
12. How can we improve the product to better meet your needs?
13. survey.io is more powerful with filtering
14. Would it be okay if we followed up by email to request a clarification to one or more of your responses?
15. Upcoming survey.io features
16. Get qualitative feedback before fit
17. Must-have % by industry
18. Ask the must-have question

Transcript

Nivi: Hi there! This is Nivi from Venture Hacks, and I’m here with Hiten Shah from KISSmetrics.

First of all, I want to thank them for sponsoring our interview with Sean Ellis and making it available, free, to you guys.

We’re going to talk a little bit about what KISSmetrics does and how it ties into the things that we discussed in the interview with Sean. And Sean is actually an advisor to KISSmetrics. Is that right?

Hiten Shah: Yes, that’s right.

survey.io: Before product/market fit

Nivi: Cool. The interview with Sean was broken into two parts — before product/market fit, and then after product/market fit, where you do the things to prepare for growth.

Why don’t we talk a little bit, first of all, about what you guys have and do for the before product/market fit stage?

Hiten: Sure.

Nivi: Take it away!

Hiten: Yeah, I will! First of all, I want to thank Venture Hacks. We’re happy to support it and thank you guys for bringing interviews like Sean and Eric and all those guys to a bigger audience. I think you guys are doing a great job of that, and that’s why we’re pretty happy to sponsor the podcasts.

In terms of pre-product/market fit we’re following a lot of these practices in our own business, so we’re eating our own dog food, so to speak — everyone says that we’re actually really doing it.

survey.io measures fit

Hiten: So that’s why we built survey.io with Sean Ellis, and the idea of it was, Sean’s got a bunch of questions he asks. The key question is: how disappointed would you be if a product name, say, Google didn’t exist? There are a bunch of key questions there. I’m sure it’s gone over in the interview, previously, in the first part.

But basically, there wasn’t a template that had that question on it. So we worked with Sean Ellis to come up with a simple product, a simple tool that was free that people can use to assess whether they have hit product/market fit, and that’s what we did with survey.io.

So anyone can go into survey.io and basically send off the survey to their users and get an idea of whether they have a product/market fit.

1. How did you discover the product?

Nivi: So the main questions are: How did you discover the app?

2. How would you feel if you could no longer use the product?

Nivi: How would you feel if you could no longer use the app — which is the key product/market fit question — very disappointed, somewhat disappointed, or not disappointed at all?

3. What would you likely use as an alternative if the product were no longer available?

Nivi: What would you use as an alternative?

What’s the thinking behind that? Do you have any thoughts on that?

Hiten: Yeah. It helps you understand how you fit against competitors, or what space you fit into. You might think that you’re a customer feedback tool, while customers might weigh you as a customer support tool. So that question is actually pretty telling in helping you understand, from your customer’s perspective, what other products they would compare your product with. So I think actually that’s a pretty important question.

Nivi: Have you ever seen any surprises on that, just to flesh it out, either with you guys or other guys?

Hiten: Yes, actually I have. In a few early questionnaires about the KISSmetrics product, an older version of it, people actually put us up against optimization tools like A/B testing tools and things like that, and we didn’t have any functionality around that. So I think that was an interesting insight. Maybe it was just the fact that the analytics space is pretty crowded and there are a lot of tools in it and it’s kind of a nebulous definition, like what is analytics?

4. What is the primary benefit that you have received from the product?

Nivi: The next question in survey.io is: what is the primary benefit that you’ve received from the product?

What’s the thinking behind that?

Hiten: The thinking around that is, basically, if you understand what benefits people see your product for and you filter it by people that would be very disappointed if your product didn’t exist, you can get a really good idea of what passionate customers consider the benefit of your product — and in their own words. So you can use that to change your messaging, and kind of A/B test a bunch of things around landing pages and things like that and see if you can have a higher conversion rate based on the messaging you have there.

survey.io is open-ended

Hiten: I’m thinking that this interview with Sean has probably had a strong impact on getting people to understand what these questions are all about, much more. And that’s what I’m hoping happens because I get a lot of questions about all of these things, as well: What does each question mean? What do I do with these answers?

And that’s actually a typical problem with most survey tools, as well. If you notice, in this survey there aren’t any questions that are very least, most, least, and have radio buttons when you go there, because you’re basically making your customer think a lot with those types of questions. With these open-ended questions people are just typing in whatever they want to say, and then it’s up to you as the person giving the survey out, to analyze the results. So, that’s a big difference with this survey and the questions that are asked.

The flock will always find the best grass

Nivi: Right. Going back to the question: what is the primary benefit you’ve received from the product? Sean quotes Vinod Khosla a lot that, “the flock always finds the best grass.” I didn’t actually understand what he ever meant by that until the interview where he… What he really means is, look at the must-have users and see where in the product they consider the grass to be — they’ve found the grass. Before that, I didn’t even really understand what Vinod meant by it. I guess that’s what he means by it. Right?

You don’t need survey.io to find the best grass

Hiten: Yeah. Let me give a different perspective. I think that’s absolutely correct, but let’s say you don’t know about product/market fit, and you don’t know about giving out this survey or asking that question because you’ve never seen Sean’s stuff. Someone who has a lot of customers, and only some of them are addicted to the product, what they typically do is go look in their logs or look in their user database and see who’s logging in the most or see who sends support requests or is contacting the company the most, and they would consider those people the flock that’s going towards being very passionate about the product.

So I think there are number of ways you can determine this type of stuff. So to me, what Vinod Khosla said feels like there are multiple ways you can determine this. This seems to be a very strong and efficient way compared to how people used to do it.

In some of our own products, in the past, we would just do it based on: Oh! This guy logs in every day. Or, this guy keeps bugging us and wants features, and at the end of every email says, “Oh, I love your product!” We’re actually getting that with our latest product that we have, and those are the people who typically you try to cater to and things like that. This puts a little more science behind it.

Nivi: Right. And this tells you why they love the product.

Hiten: Absolutely.

5. Have you recommended the product to anyone?

Nivi: The next question is: Have you recommended the product to anyone? And if so, please explain how you described it.

What’s the thinking behind that?

Hiten: This goes back to something called Net Promoter Score.

These questions are all developed by Sean, but this question is interesting because when I first saw it, it reminded me of Net Promoter Score. Net Promoter Score is a methodology that a lot of Fortune 500 companies have implemented, where they ask people: how likely are you to recommend this product? And they have a scoring system based on the responses.

Nivi: And it’s a scale of 1 to 10.

Hiten: Yeah, it’s a scale. Exactly. It’s not a simple question. It makes people think. But they have scoring around it, so I don’t know exactly how the scoring is. But if the average people say 3 to 4, and then you make some changes, you want to see those people start saying 5 to 6, or whatever it may be. And then you group… It’s not a gauge….

Nivi: Do you take cohorts of those?

Hiten: Yeah, you take cohorts of those people and try to improve them over time. And there’s a whole methodology around it. It’s pretty complex. Another part of it is….

The key thing about this question that people don’t get at first glance is that they think it’s really asking them if they’d recommend, and trying to make them recommend to people, or trying to suggest they should recommend it, and it’s actually not about that, and neither is Net Promoter Score. All it is, is that there is some psychology around if you’re willing to say you’d recommend it. You would be bucketed into more of a passionate customer. It doesn’t mean necessarily that you’ve ever recommended it, or that you would, but it’s more just to solicit how satisfied customers are with your product.

So this question doesn’t come out of NPS, because Sean didn’t even know about Net Promoter Score, I believe, when he created it, but it’s got the same kind of feeling towards it. So it’s just basically another identifier of people who are passionate.

But the real thing about asking them the yes or no question is when you ask them to describe how they would describe it to a friend, because in this case it’s an open-ended field and they’re typing in the way that they would describe it, which is basically another gauge, a different sort of angle on: what benefit have you received from the product? You’re having them actually help you figure out what your messaging should be, quite frankly.

Nivi: Yeah, and that is, literally the positioning of the product, the position it holds in the user’s mind, and that’s how he would describe the product to someone else.

Hiten: Absolutely. Yep, it’s really powerful.

6. What type of person do you think would benefit most from the product?

Nivi: Just this next quick question: what type of person do you think would benefit most from the product? Question 6.

Hiten: Yeah, I actually have a term that relates to this question a little bit, but my term is data porn. Some things are just data porn. This doesn’t fall into this category, but it just reminds me of that because I get a lot of enjoyment out of this question when I look at results, because people will describe it in all sorts of ways. You’ll hear people say: my best friend would best use it.

You know, depending on what type of product, obviously. I think this question is important because it helps you understand what your target market should be, potentially. Right?

Nivi: Right.

Hiten: And a lot of times people won’t even say it’s someone like them. They might say that it’s someone different than them, and that could give you some insight into, based on your current messaging, what people really think the type of person who would be attracted to your product would be.

So, I haven’t learned as much from this question, but it’s always fun to look at, because it’s another one of customers saying something.

Nivi: Just thinking about the interview with Sean, which you haven’t heard yet, I think this could be helpful, in particular, when the product is very horizontal and you’re trying to figure out which vertical in that group is really the one that you’re going to go after, potentially. For example, Sean talked a little bit about how they initially positioned the product in a feature-based way so as not to narrow down the use case.

Hiten: Yup.

Nivi: So here you’re basically asking them the title of the person who’s going to use the product, or what the use case is.

Hiten: Yeah. Yeah.

Nivi: So I think it can help you narrow down the positioning and the market that you’re going to go after.

Hiten: That’s a great point.

7. How can we improve the product to better meet your needs?

Nivi: How can we improve the application or product to better meet your needs? Question 7.

Hiten: That’s always a great question. People tell you…

This one’s interesting. I think this one gets the most quantity, especially with passionate people. They want to tell you exactly how you can improve their product.

If you haven’t asked this question to your customers, I don’t understand why not, because they’re going to give you all kinds of insights into little tidbits they don’t like. I’ve seen people answer this question and go into: oh, the button should be moved; or the UI is like this; or even try to get into really descriptive stuff.

survey.io is more powerful with filtering

So it’s just a cool question, and again, all these questions are really powerful when you start filtering. That’s why actually we don’t have an export feature that’s public on this thing, but a tidbit is that if you just add a /export to your results, you’ll get an export of the data.

Nivi: OK, nice. Yeah, and the filtering would actually be super interesting. The first segment filter is by must-have, nice-to-have, and I-don’t-care users.

Going back to Question 7: how can we improve the product? Again, you haven’t heard the interview yet, but what Sean talks about is going to the nice-to-have users and figuring out how you can improve the product for them and turn them into must-haves.

Hiten: Yup.

Nivi: So he gave the PayPal example, where some of the nice-to-have users may have still been using the mobile to mobile payment. You don’t care about those guys because you’ve seen like 1,000x increase on desktop to desktop payment. But of those nice-to-have users who are using the desktop to desktop payments, you might want to know how they want to see the product improve; for example, better security or if it had my address book in there already or something like that.

Hiten: Yup.

8. Would it be okay if we followed up by email to request a clarification to one or more of your responses?

And then: would it be OK if we followed up by email to request a clarification? Question 8. What’s the thinking behind that?

Hiten: These are people that you have… you know what they answered, and you can segment into different groups and you can follow up with them with follow-up surveys and things like that. It’s always good to get email addresses.

What we’ve noticed, on the weakest products, the ones that aren’t as sticky, maybe are more consumer oriented, we’ll still see 50% of the people giving an email address. And you might not even have the email address of these people because one of the things we let you do with the survey is embed it on the page, and it might not be a logged-in user so you might not even know what their email address is.

So now all of a sudden you’ve got these people, you’ve got their email address, and you’ve got all these answers associated with that email address. So that just becomes really powerful to follow up with them or, back to the previous question, if you add a feature that they want, you can go notify them and let them know you added it because they wanted it. Customers get delighted by that. I don’t see why every company isn’t trying to delight their customers — it’s one of those things.

Nivi: Right. And when you’re in the trying to get to product/market fit stage, getting a list of potential, early evangelists is probably at the top of your to-do list. Right?

Hiten: Absolutely. Yeah, we don’t even let you edit the questions yet. That’s a feature coming out. We’re building a much deeper product out of this. We have some ideas around it. We’re still working with Sean on it.

Upcoming survey.io features

Nivi: What’s coming in this? The segmentation would be cool.

Hiten: Yeah. It’s been a very interesting conversation at our own company about what we should do about this. One thing is we want to, obviously, have this same thing for free, but we’ve also got some ideas on how to get qualitative feedback from customers in a much leaner way where you almost ask them one question at a time. You might build up some of these answers over time. So we’re working on embeddable surveys and things like that, to do that, and this is stuff we really haven’t seen anywhere else.

Also there are things like, if you know someone’s username or email address, you could already pass in to the survey so that you don’t have to ask for it. And then you can associate a specific customer with the response, and stuff like that, which SurveyMonkey and other tools currently make a little bit tedious to do, or basically impossible in some cases.

We don’t want to compete with SurveyMonkey or the survey tools, so we’ve really thought long and hard about what we want to do about this. We will be launching something, probably by mid-to-end-of January that’s a whole revamp on this tool. It makes it so, hopefully, you don’t have to copy and paste the questions into other tools. So editing, filtering — we’re really focused on embeddable surveys. These are just some of the pains that we’ve noticed about getting feedback from our own users.

If you ask them one question within the interface, it’s really powerful because you can get a lot more context. So in our system, if they’re looking at an analytics report, we might want to ask them: is this report useful to you? And if they say yes, say great, thanks! We love you. If they say no, we can pop-up a form field and ask how we can improve it for them. Those are the kinds of more mini-surveys and mini-questions that we don’t really see a product online that does that, so we’re going to move the product in that direction.

We’ll always have this specific survey for free though, and add some of that stuff. And this is all about….

Get qualitative feedback before fit

We feel that before product/market fit, what’s really important is those early customers and getting as much feedback as you can that’s qualitative from them, and quantitative metrics aren’t as important as the qualitative metrics at that time.

Must-have % by industry

I’m amazed by some of the percentages I see on the disappointed score that’s segmented by industry. In general, trends-wise, if it’s a SaaS sort of product, like a web app, I tend to see a higher score on the percentage of people that would be very disappointed.

Nivi: So that would actually be a great little study for you guys to publish. Right?

Hiten: Yeah. And another thing, on the opposite end, is that we notice score in the 20% to 30% for consumer applications. Typically they’re very low. People wouldn’t say that about Facebook, necessarily, but a lot of people would say that about more trivial consumer sites. I would say that you would probably see a lower score on something like YouTube, where it’s very passive and people aren’t necessarily very passionate about it and there are a lot of alternatives and there isn’t really some sticky glue that makes people really want to come back. So those scores are usually under 40%, while SaaS apps I’ve seen as high as 80%

Nivi: Well that makes sense, especially if you’re going to filter by… SaaS guys are already paying for the product; they’re probably going to consider it to be a must-have.

Hiten: Yeah, and even some of those freemium businesses though. Think about the emotional attachment to a free product you have if you’re constantly using it. You’re going to be disappointed if it doesn’t exist. Right? So I think those are the trends that we see, overall.

Ask the must-have question

Nivi: The key question here, if you’re going to walk away with one thing from survey.io, is that you should probably survey the people who actually use the product and ask them how disappointed would they be if they could not use the product anymore.

Hiten: Yup! Absolutely.

To be continued in part two of our interview with Hiten Shah, where we’ll talk about their product called KISSmetrics, a new tool for optimizing funnels.

Music: Squarepusher

If you don’t know Sean from his blog or tweets, he lead marketing from launch to IPO filing at LogMeIn and Uproar. His firm, 12in6, then worked with Xobni (Khosla), Dropbox (Sequoia), Eventbrite (Sequoia), Grockit (Benchmark)…  the list goes on. 12in6 “helps startups unlock their full growth potential by focusing on the core value perceived by their most passionate users.”

This is the first time Sean has done an interview on the record. I’m really psyched he’s making his insights public — this interview is a must-listen. We’ve broken the interview into two parts: 1) before fit and 2) after fit. This post contains Part 1 (and here’s Part 2.)

SlideShare: How to bring a product to market
Audio: Interview with chapters (for iPod, iPhone, iTunes)
Audio: Interview without chapters (MP3, works anywhere)
Transcript with highlights: Below

This inteview is free — thanks to KISSmetrics

We’re bringing this interview to you free, thanks to the kind support of KISSmetrics. Sean is an advisor at KISSmetrics and we interview their CEO, Hiten Shah, in How to measure product/market fit.

KISSmetrics built survey.io with Sean — now they’re collaborating on KISSMetrics, a new tool for funnel optimization.

Prerequisites

You’ll get more out of this interview if you also read:

1. An example of the survey.io survey Sean uses before fit. (Several phrases we use in the interview mean the same thing: Product/market fit = Fit = 40% of surveyed users consider the product a “must-have” = 40% of surveyed users would be “very disappointed” if they could no longer use the product.)
2. The startup pyramid.
3. Some of our favorite posts by Sean.

Outline

Here’s an outline and transcript of Part 1.

1. Half the marketing battle is the product
2. Must-have products make marketing much easier
3. How PayPal built a must-have product
4. Understand the must-have users
5. How to get to product/market fit
6. How not to get to fit
7. Should I launch?
8. You can’t set a deadline on fit
9. How to communicate with the board during fit
10. How to use positioning to improve fit
11. It’s frustrating to try to grow without fit
12. How many users do you need to determine fit?
13. Who do you survey?
14. What is promise?
15. Don’t over-position your product during fit
16. Pivot the business around the love
17. Focus the product on the love
18. Create a great experience around the love
19. Create a business model around the love
20. First find the love
21. Are recommendations a good indicator of fit?
22. The must-have metric is a good indicator of fit
23. Preview of Part 2: What comes after fit?

Transcript

Nivi: Hi, this is Nivi from Venture Hacks, and I’m here with Sean Ellis. We’re going to talk about how to bring a startup’s product to market.

This interview is broken up into two parts: before product/market fit and after product/market fit. Or to put it another way, what are the set of activities you have to do to get the product/market fit, and then once you’re there, what are the set of activities you have to do to prepare for sustainable growth.

I’m super psyched about this interview. This is Sean’s first-ever interview on the record. We’re going to jump right into it.

Half the marketing battle is the product

Sean Ellis: The whole idea that’s gotten me to where I am right now, where I’m doing multiple companies, is basically that I’m just trying to get lots of cycles, up front, in that really critical zone — I call it the fail zone — that if you don’t get that done right, the company will fail. And if you figure that part out, then it’s really a question of how much success is the business going to have?

Particularly, one of the things that I’ve learned that’s been a little humbling through the experience is that if nobody wants the product, it doesn’t matter how effective the marketer is, you’re going to have a really hard time being successful. And if people want the product, you don’t have to be that great a marketer — the product is just so much easier to market. So half the battle, I’ve realized over time, is just finding the right company with a product that people actually need, and that there’s a big enough market that can support growth for that business.

Must-have products make marketing much easier

Nivi: Yeah, and what might be interesting for some people to hear is what’s in that grey zone between nobody wants the product, to a lot of people consider the product to be a must-have. For example, sites that have great marketing and distribution through viral marketing, but they don’t necessarily have any real must-have-it-ness with their customers or with their users.

Sean: All right, so the question really being: the grey-zone products that fall between those must-have products, and products that maybe aren’t as strong of a must-have, but can you be successful with those?

I am just figuring this out as I go along. I know that there is no right answer on any of these things. What I do know is that if you have a must-have product, your chances of success are very high.

And the way that I figure out if a product is a must-have or not is: what percentage of people are telling me that they’d be very disappointed if they could no longer use it? But it seems like you find a lot of categories where you might say that it’s a nice-to-have, rather than a must-have.

And I think that the difference is that for a nice-to-have product — so, something where not a lot of people say they’d be very disappointed without it — that you probably can be successful, but it’s going to take a lot of effort on the marketing side to make up for those product deficiencies.

I think if you really step back and say, as a business, that ultimately our goal is to create a successful, fast growing business, that people really need our product, and that over time we have a defensible business where it’s really hard for somebody to come in and provide an alternative product where we start losing customers, that in an early stage startup you have an opportunity to put a product out there, get that initial user feedback, and then you have two choices.

If that feedback is really strong and the users say that they’d be very disappointed without it, in a large percentage, then you can try to grow the business.

Alternatively, if they come back and say that they wouldn’t be that disappointed without the product, then you have the choices where you can either try to grow it or you can decide that you’re not going to try and grow the business. And to me, my recommendation has always been to decide not to grow the business if it doesn’t have a lot of people that are real passionate about the solutions.

Actually, you can operate a startup at a pretty low cost if you basically have a small engineering team and you’re putting product out there and you’re not spending a lot to get people on there, but you’re really engaging the people that do come in. You can engage them much better if there aren’t that many people, so you’re in a position where it may take you a year or two years, but you can really, over time, start to evolve a product into being something that someone wants.

Maybe you find early on that absolutely nobody wants the product, and in that case you might want to do a complete restart on the business.

And when I’m presenting to groups, the example that I always point to, mostly so that people know that there’s hope if they don’t have great product/market fit initially….

How PayPal built a must-have product

Sean: That product/market fit is how I describe having a large percentage of people that would be very disappointed without the product. If you’re above 40% of the people saying they’d be very disappointed, I tend to say you’ve found product/market fit, and if you’re less than that, you haven’t.

So a lot of times I meet with companies that are maybe in the 15% to 20% range on that, and it’s very discouraging. You’ve worked very hard to put out a product, so the example that I give them is PayPal, where the initial product that they came out with was a cryptography product where you would use a PDA to store access codes for servers, and it was better than carrying a bunch of individual devices to access those servers. And the problem they were solving was that there was a lot of clutter in carrying those devices, and they were going to consolidate them all onto a single hand-held.

In “Founders at Work” there is a great case study on this. What Max Levchin says in “Founders at Work” is that they put the product out there and “nobody really needed it.” That was a direct quote — “nobody really needed it.” So what they decided…. Most companies would try to interpret the data and keep blasting away and say that somebody needs it. They just don’t get it yet.

Nivi: We need more distribution….

Sean: Yeah, they blame marketing, they blame sales, but ultimately, it took a lot of guts, and just honesty, to say: You know what? We created something that nobody really needs.

So what they did at that point was they stepped back. They asked what are we good at? We’re good at developing for a handheld PDA. We’re good at security. What else can we do with these skill assets that we have? And they decided that they were going to get into the mobile PDA payment space, where basically, people could beam payments back and forth.

And on the initial vision they were able to raise a couple hundred thousand dollars. On this new vision, they were able to get a prototype out on that initial money that they had raised, and they were able to raise… I think it was four million dollars. I don’t remember the exact number on it. And they actually beamed the VC money over so they had a good enough prototype to be able to accept the payment via this PDA payment. And then they were off to the races, and they started executing the business.

I think, in that case, my guess is that if they had surveyed those users who were using PDAs to make payments; it was more of a party trick. It was something that there wasn’t a real big need for, but they got users on there. They got lots of users on there, and they were able to get those users coming back. But, I’d put it in that probably-nice-to-have category.

Nice-to-have is kind of a scary thing, because you get enough success that you want to keep at it. But they were very fortunate in that they sort of stumbled into what ultimately became their market, which was, of course, the web-payment platform that PayPal is today. And that was really a function of people just starting to use it that way. And so, that’s why I focus a lot with companies on helping them figure out the use cases behind their product. If the people who have one use case consider it much more of a must-have than people who have a different use case, then that’s probably the business you’re going to be able to build a successful business on.

Understand the must-have users

Nivi: So, do you ask the must-have users what they use the product for?

Sean: I ask everybody what they use the product for, and I’m trying to see, really, who has the most passion around the product. And usually it breaks down by use cases, sometimes by user types. It’s really just looking through the initial data to really understand who really needs this product, and it can be on demographics, it can be on use cases, it can be on a lot of different things. But once you really understand a group that really needs the product, then you start to have that true North for the business — the part that you can actually start to build a business that will grow and thrive around if they represent a big enough market.

Nivi: Right. In that product/market fit stage that you’re talking about, which is basically the bottom layer of your pyramid, what don’t you do? For example: PR, trying to get a lot of distribution, increasing your burn.

Sean: Well, it depends a little bit on the type of business that you have.

So just one quick thing on wrapping up the PayPal example was, by the time they pulled the plug on their mobile-payment business they actually had, I think it was 1,000 times more people using the web payments. They had obvious traction, so they didn’t need to necessarily get that from a survey. They could just see that they had the traction there at that point. They could say: You know what? That’s where our real business is. Even though they fought it, initially, because they had different preconceptions about where their business would be.

How to get to product/market fit

Sean: I purposely try to avoid working with companies that are pre-product/market fit, because they’re in such a risk zone that, one, it’s going to hurt my reputation if I string too many of those together, and two, to ask people, who have a product that people don’t need, to pay me to help them with marketing would just not be the right thing to do. It would be a mistake for them to pay me, and it’s just not smart, from my perspective, to stake my reputation on their success.

But I found myself working with one fairly recently, on a very short, almost advisory project, of just helping them out. And my advice to them was to really be laser focused on that number and to really monitor that very-disappointed number.

So, one of the things that I’ve seen work well for companies that are in the 20% range, or even lower, of the user saying they’d be very disappointed without it, is to focus on, one, why those people would be very disappointed without it, and to really start to say, OK, this is our best signal of value that we’re creating.

And then, to look at the feedback from the people who would only be somewhat disappointed without it. Ignore the people who say they would not be disappointed without it, because they’re so far from being satisfied.

But then look at those somewhat disappointed people, and look at feedback from those people — particularly on what changes they’d like to see in the product. And look for things that relate to the very-disappointed feedback. If you take all feedback equally, you’re going to have a very broad product experience that is very disjointed.

And I think one of the mistakes when people are too responsive to user feedback is that it’s a product that does everything, because if you try to please everybody, you’re essentially going to have a very unfocused product. But if you use the feedback from your very-disappointed people to give you that focus, and now you’re looking for feedback from everyone else that relates to what the very-disappointed people have told you…

Nivi: Like with PayPal, for example.

Sean: I think, using PayPal as an example, if they had surveyed all of their users, I’m assuming that a lot of the people who were doing the hand-held payments would basically have been saying that they’re only somewhat disappointed without it. And then the people who were doing the web payments would be saying that they’d be very disappointed without it. But some of the people on the web payments would be saying somewhat-disappointed, and they would say: if only I felt more comfortable about security; or, if only I could do it faster; or, if only you could give me better records of payments that have been made.

So you could take the feedback from the somewhat-disappointed who were using hand-held payments, and act on that to try to get there, but then you’re not really honing in on where the passion is.

Or, you could take it from the other side, saying, OK, these people may be a little pickier about what defines something that’s a must-have, but the feedback that they’re giving would improve the experience for the people who already say it’s a must-have. So, it’s really trying to just hone a consistent, core-product vision that is already starting to crystallize based on the feedback from the really passionate users.

Nivi: Yeah, that’s great. So in this case, it’s related in the sense that somewhat-disappointed people are using the same segment of features in PayPal.

Sean: Yeah. It’s that use case.

Nivi: Use case.

Sean: Yeah. They’re suggesting improvements to the use case that’s consistent with the most gratified users there are.

How not to get to fit

Nivi: Great! Yeah. So that’s a great idea in terms of getting what you need to do to get product/market fit, and I’d like to talk more about that, but it would also be great to hear what you don’t do.

Sean: I think the thing that you don’t do — and I’ll make one caveat in a second, on this — is that you don’t aggressively try to grow the business. Particularly, you’re not trying to do business development relationships. Maybe you can spend enough to create some flow so you actually get people giving you the feedback on the product, but one thing I wouldn’t do is obsess over ROI on that flow.

Your goal is not to figure out that repeatable, scalable customer acquisition engine at that point. Your goal is to get enough people in there to where you can react to their feedback and hone the product experience on their feedback.

Should I launch?

Nivi: How about a launch?

Sean: This is very debatable. A lot of people would say that a launch is a really important event. For me, I’ve never been big into launches. Launch is a one-off. With launch, you’re going to get a group of people in one time, and you’re going to get some feedback on them. And yes, they may be cheaper because you got some good press around it, but you haven’t learned anything about how to grow the business with a launch. So for me, I’m much more….

I wrote a blog post way back when. I think it was called “Launch with a Trickle,” and that’s much more of my focus on things. Just get enough people on there as early as you can, so that you can react to their feedback. And you may find that they absolutely love the product. Then hone in on why they love the product. And then get the messaging right and the experience right, so that you’re delivering that better. Once you’ve kind of tightened up and you’ve got that validation that a lot of people love your product — hopefully more than 40% say they’d be very disappointed without it — then you can step on the accelerator and a big launch might make sense at that point.

You can’t set a deadline on fit

Sean: But what I see too many companies do is set a date. They say, we’re going to have our launch on January 14th, and they’re two months away from that and they’re working towards… I mean, it’s either time or it’s not time.

Nivi: Yeah, and you can’t set a deadline on product/market fit. Right? That is the point in time, in the growth of a company, where you basically are unable to predict anything.

Sean: Right. And all you’re going to do then is make the mistake of being too aggressive too early, or piss off your investors by missing that date for the launch.

So it’s much better to say, our clear objective — the next milestone that we need to get to before we do all of these things — is to actually make sure that enough people love this product. As soon as we get there, then we have to figure out why, and make sure that we’re really honing every piece of the business to reflect why people love the product.

How to communicate with the board during fit

Nivi: Right. So, on that topic, how do you communicate with the board and set expectations when you’re trying to get to the product/market fit stage?

Sean: What I’ve found is that people either get it or they don’t.

Nivi: What are the responses of the people that don’t get it, like from a board perspective? What have you heard board members say directly, or secondhand?

Sean: They’re just: You’ve got to go for it! You’ve got to go for it! It’s good enough. It’s good enough. And they just want to accelerate the business.

I have a great example of a company that was about 15% or 20% on product/market fit and I loosely advised them at no cost, because I was going to spend some time in Scotland and they’re based in Scotland. And I wanted to get to know these guys better because I was going to be spending a week over there. So we just worked together over a period of time.

And when we first started working together they were at the 15% or 20% very-disappointed number, and they explained to me that their board of directors was just pushing them on: We’ve got to get to this growth number! We’ve got to get to this growth number!

And they took the time to really explain: This is our goal right now. We don’t think our product is good enough to accelerate this business. Our goal is to take the feedback from the users that we have and keep getting our product better.

In the process of doing that, they had the very painful process of actually having to lower projections on growth, and doing it week after week, and frustrating their board of directors through that process. But they did exactly the right thing, because today their very-disappointed number is over 50%.

And when I met with them when I was in Scotland about a month-and-a-half ago, they were at a really exciting point where they said they keep going back to their board of directors and saying: Hey! We’re going to revise those numbers up. They kept going and saying the numbers are too low. It’s going to be better than that, because now that they have the product right they could really focus on that, and they weren’t going to have the….

You can grow when your product/market fit is lower than that, but you’re going to grow a lot slower than you otherwise would, and you’re not going to address the real problem. And you’re going to have a very frustrating experience trying to grow that business over the next several years, versus saying: Let’s keep our costs really low, let’s get the product right, and when we’re focused on growth, no excuses. Now we’re absolutely accelerating that business and it’s going to be much easier to grow the business at that point.

And ultimately, you take it a year out or two years out, you’re going to have a much bigger userbase doing it that way than if you’re trying to grow through the whole period of time.

How to use positioning to improve fit

Nivi: Now, what changes should they make to get to that number, without getting into the specifics of the business?

Sean: I don’t know. I didn’t work with them, so all I know is that they did the things that I’ve mentioned, where they really took the feedback of the very-disappointed people, really tried to look for the signal of who loves this product. Somebody loves it. Who loves it? Why do they love it? How can we really put a stake in the ground around that and try to make that piece better, and try to play down the other pieces?

Nivi: Then maybe reposition?

Sean: Yes. Some of it’s definitely through repositioning.

I have another company that I worked with that was before I had [done] that. This was really the company that helped me come up with the idea that I was going to survey people ahead of time, because I got into two companies at the exact same time who were both at about [where] 7% of their users said they’d be very disappointed, and found myself in the “oh, crap!” moment of figuring out that now I’m going to have to help these guys transition to growth and grow, but they’re not ready for it yet. And why did I wait until I had a six-month contract with them to run the survey? From now on I’m going to survey it ahead of time.

But at one of those two companies, we were able to see a really strong signal in the 7% that said they’d be very disappointed without it, reposition on that signal… [interrupted]

Nivi: So how do you reposition on the signal. In this case, did you get a little bit more data…?

Sean: We just found out — I mean, I can’t give any specifics on it — they had a product that had lots of different features, and each one of those products could be a product in itself. They built a really big suite of tools, within a product, and everyone who said they’d be very disappointed was focused on just one of those. So we basically only talked about the one they were focused on, and we didn’t talk about any of the other things.

Nivi: To the user, say, on a landing page or on the home page.

Sean: Exactly — on the home page we only highlighted that and said we’ll introduce the other stuff after we get them in. One of the benefits with that was that we were really able to simplify messaging. Having a really complex product set that you have to present people is way harder than simplifying that and only highlighting one. So, we basically highlighted the most important must-have product from the group, and hid some of the other products, and as a result we could be much clearer in what the benefit was for the people coming through and what we were actually offering.

So, one week later, on the next cohort of people coming through, we were over 40% because they had a very different expectation of what this product was, just by changing the messaging and focusing their first user experience on just this one piece of the business.

Nivi: Right. So, with no real change in the product, but a change in the messaging and a little bit of first-user experience change, you got a huge uptick in the must-haves.

Sean: Yup, exactly. And the interesting thing is that business, probably out of every business that I’ve worked with, is the one that I think has the opportunity to be a multi-billion dollar business, and one that I’m most excited about. And when I first saw that 7% number, it was the one that I thought: how do I get out of this? So now I’m really happy that I stuck with it, and I’m happy that we were able to figure those things out. And I’m really good friends with the founders.

It’s frustrating to try to grow without fit

Sean: When everything’s going well, the chemistry gets much stronger and it’s just a much more enjoyable place to be. That’s just one of the things that I’ve learned, is that in a company with a low product/market fit, it’s not a fun place to be. It’s frustrating trying to grow that business.

And for a company that can really embrace the fact that: You know what? We missed. The product that we created just isn’t something that people really seem to want or need.

Nivi: And we don’t really measure ourselves through growth right now, either.

Sean: Yeah. So for us, we’ve got a lot of money in the bank. Somehow we got lucky enough to raise the venture capital. If we try to grow we’re going to waste a lot of that money trying to grow. Let’s just keep our costs way down, get just enough users in there and either do a complete reset on the business — like PayPal did, which I think most companies don’t need to do — or just find the love.

In my last blog post I talk about “just find the love.” Somebody loves it. Somebody really needs that, and once you find that, that’s the stake in the ground that you can figure out how to create a successful business.

How many users do you need to determine fit?

Nivi: And what is just enough users?

Sean: That’s a good question. I think it kind of depends on the business, but ideally, if you have a product that ultimately is going to cost thousands of dollars, then it’s just not going to be realistic that you can bring in a hundred every day on that product. But if you have a product that’s going to have a 10% upgrade rate or a 10% purchase rate, and it’s going to be $100, then that’s a product that you probably want to try to have a pretty constant flow of around 100 new people a day.

And whatever it costs to get to that 100 new people a day is a really good flow to iterate on. If you can only be at 30 or 40, that’s probably OK.

Nivi: And how about on the output side of that — the people that you’re going to survey, the people that have gotten to the finish line? How many people do you need to be talking to, to get a good signal there?

Sean: I look for around 30 responses as a minimum. It really depends. If you have a very low percentage of users that say they’d be very disappointed… Say you’ve got 30 responses and you’ve only got 5% that say they’d be very disappointed without it, that’s not a whole lot to work with on finding a signal.

But I ran a survey this morning with a business that has 70% of their users that say they’d be very disappointed. We got a very strong signal on 42 responses.

Nivi: OK. Right.

Sean: Really, it kind of depends, first, on how strong that number is. And then what you’re trying to do is get enough responses in there to find some sort of signal that either says: We completely missed, or that [there are] very few people we’ve hit with, so we want to make sure that we survey enough people that we’ve got maybe 10 or 20 people who say they’d be very disappointed, that we can start to hone in on the feedback that they’re giving.

Who do you survey?

Nivi: And do you ever have trouble defining what that finish line should be? For example: active monthly users, people who just signed up for the product, or is it… [interrupted]

Sean: Do you mean on who to survey?

Nivi: Yeah.

Sean: What I tend to do is say, people who have been back at least once.

And I think for a consumer product you shouldn’t be charging at this point; it should be free at this point, because the money side already starts to put a big filter out there. Even if you had a trial, people would make up their mind pretty quickly. If they decide they’re not going to buy it, then they’re not going to say they’d be very disappointed without it, and not buy it. So, just pulling money off until you can get that signal is pretty important.

For me, I define what is actually the experience that we think is the experience that gives them a full taste for the product, then I want to make sure that people have at least done that when they come back to us.

Nivi: So in the case of PayPal what would that be?

Sean: That they’ve at least made a payment with PayPal or accepted a payment.

Nivi: One payment?

Sean: Yeah. And then, that they’ve been on the site within the last two weeks.

Nivi: OK.

What is promise?

Sean: So those would be kind of the two pieces. You just asked about intent and the value proposition in some of those things, so I think the important starting point is that the promise or the differentiated value proposition is really based on whatever signal that you find. Where you find the love on the product, that’s what you’re trying to do there.

Nivi: And you call that promise.

Sean: Promise is one of the names I use for it, but it’s all synonymous with just the value proposition, the differentiated value proposition of just what is this product going to do for you. What is the unique piece about it?

Nivi: Right. So in the PayPal case: send money to friends online.

Sean: Yeah.

Nivi: OK.

Sean: Yeah. I think that would be… And at the time, that was fairly differentiated. Now you can use credit cards and other things, so something that remembers your log-in information might be trying to differentiate from just Visa or Master Card or American Express.

Don’t over-position your product during fit

Sean: But what I was going to say with that is that there are two periods where that becomes important. One, before you have product/market fit you’re trying to find that signal, so you can do some sort of early positioning to try to highlight that. But the problem is, too much positioning is going to steer your users in a certain direction. So, early on, a lot of times it’s good to actually be very feature-function oriented.

Nivi: OK.

Sean: Because unless you have a real strong signal…. You can experiment with it, like, OK, let’s bring people in and we’ll give them this message and we’ll see what their very-disappointed number does, just on this group of people. We’ll survey them separately.

But in general, if you don’t have a real strong signal, then it can be kind of dangerous to shape their experience by highlighting some unique attributes that turn out not to be the important unique attributes.

Nivi: Because it might be inconsistent with…?

Sean: Because you’re going to shape their experience so much that they may miss the other thing that would have been the thing that would really thrill them.

Nivi: Right. So give me an example of how I would do that.

Sean: X Corp. is a prime example on that. In the early days at X Corp., it was just one of those things that I realized that if I really focused on homepage messaging it was going to take me a really long time to get some level of consensus, and that’s just the way that the business….

The way most startups operate, early on, is that everyone has opinion around what the homepage looks like and what it says. And so, what I realized is that getting a lot of those early users, we were going to be getting them through search anyway, so I basically said I’d let everyone else figure out what’s going to be on the homepage, I’m just going to take people in through landing pages.

With landing pages you know what the person’s intent is when they’re coming in through a specific search word, so you can really hone things in a more segmented way.

But our homepage ended up being very feature-function oriented, and the group we initially built the product for ended up being a relatively small percentage — especially on a revenue side — of where the business ultimately ended up generating a lot of revenue. So I think if I had taken a traditional marketing approach to that I would have been very benefit-oriented right from the beginning. And I probably would have positioned it away from what ended up being the really interesting market for us, because I would have given so many details about how they should use the product that this other group may not even have considered using the product.

And so, as a result of having, essentially a group of engineers be able to initially determine the wording for the homepage, they just focused on the features that they created, with not a lot of why someone would need it. And that turned out to be the best thing that could happen because, ultimately for us, by the time I really started looking for that signal, the signal was very different than what we would have initially had if I had positioned out of the gate for that.

Pivot the business around the love

Sean: A big theme that I keep coming back to is, where is the love? It’s basically just finding out which users are passionate about the product. What is the use case on the product that those people have? Why do they actually love the product? And that gives you a good core of information to guide every part of your business. So messaging, you want to reflect that, but also, now that you know that, you want to make sure that you can really get a lot more people to that particular type of use case and that particular type of gratifying experience. So understanding user gratification is really critical in all of it.

First of all, you want to know that you have enough that you can try to grow the business — that you actually have something that will be growable. And then in growing the business, having that gratification should give you guidance on everything. So product roadmap should be a function of how people are gratified with the product and who’s gratified, so it should give you a lot more focus on that product roadmap as opposed to a lot of feature feedback and ideas and different things that, over time…

You know, initially you have to kind of scatter-shoot it. You have to cast a fairly wide net of things that might please people, but once you really know where that gratification is, that’s where you really want to double down, and that’s your business. Before, everything’s an experiment. But your business is really on where that gratification or where that love is.

Nivi: Does that go to what is the primary benefit of the product in your Survey.io?

Sean: Yes. That’s where I start to see it. I start to see it in each of those questions. I can start to see, in the very-disappointed question there’s a: why did they pick that? You can start to see it in there, as well.

Ultimately, that’s the first thing that I’m trying to look at. One, do enough people love this product? And two, why do they love this product? And then everything that I’m doing is basically helping the business to — I wouldn’t say reposition, because reposition is associated with messaging — realign itself around that really strong user gratification. And once you have that alignment around that, it gives you a lot more direction.

Early on, everything is an assumption, everything is vision, but once you have users on there you can finally start to find that nugget of value.

Focus the product on the love

Sean: And one of the things, if you really step back and ask a lot of entrepreneurs why startups fail, a lot of people rightly focus on lack of focus as being what causes a lot of startups to fail. But then the question is, OK, if it’s lack of focus, what should you be focused on? Obviously, lots of focus on the wrong thing is still going to cause you to fail, so then it’s a big question of what you should be focused on. And for me, ultimately it’s that strong signal that this is something that users really love about the product. So understanding who loves that product — who really has that need — that starts to give you guidance on who you’re going to target going forward.

What other things can we do that can enhance that use case — that can make that particular use case? How can we prioritize those feature enhancements, the parts of the product roadmap that relate to that, and maybe drop some of the things that are the wild cards that we’ve had in there for a while?

The thing that amazes me is a lot of startups have got a little side-product project of a completely unrelated product or something else, which I think, early on when you don’t have product/market fit, is probably not a bad idea. You’re throwing a lot of stuff out there to start to see what sticks.

But once you’ve found that signal, you’re ahead of 90% of the startups out there. So once you find that strong signal, your ability to be really successful is focusing on that strong user gratification and passion and figuring out how to get more users to it, more of the right types of users to it. How can we make it even better — but, that part better, not all parts better, that part better?

Create a great experience around the love

Sean: How can we completely refine that brand experience?

Now is the time to go and get a better designer to come in and really get the graphics perfect on that part. Now is the time to bring in someone who is the absolute wordsmith to get the descriptions perfect on the website, but making sure that they’re being very responsive to that user feedback and they’re not just being a wordsmith by what they think is what’s important to people.

Create a business model around the love

Nivi: Business model?

Sean: Business model is absolutely a function of that. You want to make sure that you are not trying to monetize something that’s not important to users. If users tell you they’d be very disappointed without it, the other thing that they’re telling you is that you could keep cranking that price up for a long time before they change to being somewhat disappointed without it.

And if you’re really focusing on that to determine your price and how you charge for the product, you’re going to have a lot more signals.

First find the love

Sean: So I think that’s a big theme that startups should look at, is focus, and focus on what and when.

The “when” — early on your focus is on just trying to find some love within your product. Find somebody who loves something about your product. And you can get that day one, before your product is even developed, just by talking to people and saying: Hey, this is the problem that we think people have out there and this is how we’re going to try and solve that. Does that seem like something that would be useful for you?

Nivi: The customer development approach.

Sean: Yeah, the customer development approach. Steve Blank has a really good book that’s documenting everything on that and has written a lot more about that.

I purposely, again, have stayed out of that area because it’s so high risk, and you can’t be good at everything so that’s the part that I’ve said I’m not going to try to be good at.

But essentially from day one, it’s all about that search for something that somebody really needs.

Are recommendations a good indicator of fit?

Nivi: Going back to the must-have — that metric — you don’t see user recommendation in that metric there, as well?

Sean: It’s interesting; I see a lot of correlation on success in businesses with the must-have.

I had somebody contact me recently who had a 70% must-have number and they are adding 20,000 new users a day with great monetization, and they just started recently. It was just one of those things that wasn’t surprising to me to see all that success because they have that strong must-have number without really even having a marketer in the company.

So the must-have, I think, is really big, but there are a lot of people that have good recommendation numbers that struggle, and part of that is that… I think YouTube videos is a prime example that sometimes you just see some really funny things that you want to share with other people, but ultimately would you be very disappointed if you couldn’t? Probably not.

Nivi: Right. So it could be a false-positive. But let’s say for the folks that do have the 40% must-have, do you see a correlation of recommendations then?

Sean: Not necessarily. Sometimes I see it really high. A company that I surveyed today, with surveys rolling in right now, was 90% recommend rate, but sometimes I see it being pretty low on a company that’s doing really well. Sometimes I see, as I said before, companies that are pretty low on the must-have, but high on the recommendation rate. So I actually don’t think that’s a good proxy for success, but I haven’t studied it enough to know for sure.

The must-have metric is a good indicator of fit

The only thing that I do know is, on the must-have number, I’ve probably looked at 150 companies now on that number and I could maybe point to one or two that were less than 40% that had any kind of traction and success. And over 40%, everybody had some success. And maybe it’s 35%, maybe it’s 45%, but somewhere around there you get enough people.

The only thing I tend to find a challenge with is that some companies have a must-have number that’s around 40%, but then the feedback is all over the board on why they consider it a must-have and that becomes really hard for being able to find any kind of signal to base the business on. So that’s a different problem, and I haven’t figured out how to deal with that.

Nivi: There are too many segments in the customer base.

Sean: Yeah. Everybody loves it, but for different reasons. Then maybe you have to figure out which one of those reasons represents the biggest target user group and hone in on that.

What comes after fit?

Nivi: Right. One thing you talk about is, once you have the product/market fit, trying to get through the next few steps as quickly as possible. So it would be great if you could talk about that, and also, for people that don’t know what the pyramid looks like, what are the next few steps?

Sean: Sure. As I said, if you don’t have product/market fit, you want to be obsessively focused on getting there, however you’re defining product/market fit. I think the easiest kind of definition to work toward is trying to get 40% of your users to say they’d be very disappointed without it.

And during that period you’re trying to stay very low burn and very conservative on all of your expenses within the business. And once you find 40% of your users that say they’d be very disappointed without the product, then you’re in a position that you have a business that can grow now. So then the question is: do you try to grow the business right then or are there some things that you can still do that are going to make it even easier to grow when you’re focused on that? And what I’ve found in working with so many companies at this stage is that you’re definitely better off waiting a little time until you try to grow the business.

Nivi: And what does it mean to grow the business? You’re talking about spending money to acquire users, essentially.

Sean: Acquiring users, for a lot of businesses, means starting to spend money, but it might also mean really trying to crank up the virality of the business, or it might be SEO that doesn’t require a lot of spending.

Nivi: But it’s time and money that you require for people… [interrupted]

Sean: Yeah, and focus. The time piece, you could say, has time or focus. But it’s essentially saying…

Music: Tortoise