As promised, here is my 20-min presentation to the Lean Startup Machine event on July 23, 2010. Regarding the event, I was pretty impressed with how much the teams accomplished over the weekend, and their willingness to get out of comfort zones.  This deck is neither as pretty nor as good as David Cancel’s recent talk, but it served the intended purpose of giving the participants a taste of customer development put into practice.  Below is some quick color commentary on the slides:

2. Who am I? Key message: all of the previous startups I’ve been involved with talked to their customers, but that isn’t enough.  “Lean startup” is about ruthless and rigorous testing of assumptions and hypotheses. Think of your startup as a giant stack of assumptions — the more you push testing those conscious or unconscious beliefs into the future, the greater the risk they will come and bite you. I wrote more about previous experiences and lessons here.

3. The Context. Our particular decisions came out of the context of Aprizi’s customer, product, business model, etc. With that in mind…

4. The first rule of lean startup is…

5. There are no rules. Lean startup is a methodology and framework of ideas, not a playbook or roadmap. Use it to add self-awareness and rigor to your business, but make your choices based on your own context.

Aprizi’s Customer Development Steps (at which point I gave summaries of topics covered in the following older blog posts)
6. Initial Idea Validation. See this post
7. Customer Development. See this post
8. Alpha. See this post
9. Open Beta. See this post

12. Doing a Customer Development Interview. I wrote specifically about this slide here

13. Learning Comes in Waves. Focus on learning, not quotas. What I have found is that with each new step (idea, paper mockup, alpha, beta), we had an initial high burst of learning around some big decisions, and then learning started to become incremental. At that point, our efforts in customer development slowed down because we needed to make decisions, then get to work on the product so that we could get to the next wave of learning.

14. Strip Out Everything But the Core Value Proposition. On Sunday I tweeted “Many features in a young product is often a sign of not understanding the problem or customer well enough.” I was laughing at a ridiculously over-featured early Aprizi wireframe shared with the participants as an example of Balsamiq.  Needless to say, we took a serious scalpel (or hatchet) to the feature list, and of course customer development helped us focus. We simplified and tried to hone in on the point of maximum delight. When you have extremely limited resources, simplification and prioritization is a necessity, but it also just makes for a better product.

I also gave an anecdote from the very start of Aprizi. We built some code to automatically gather shopping preferences and history from a webmail account. We realized that it had nothing to do with testing the actual value proposition to the user, so we quickly tabled it.

15. Pre-PM Fit. It’s All About People. In the very early days, don’t get lost in surveys and A/B tests and optimization or even continuous deployment. That stuff is really important but gets more so as you approach and pass product-market fit. Initially, focus on people. Watch them use the product. Listen to their tone of voice. Watch their facial expressions. Remember, all business is about people. They will use your product, share your product, buy your product, or ignore your product. You need to understand why.

My last message about people was a reminder not to completely ignore business relationships that you will need later on. A bizdev deal, a PR relationship, a blogging strategy, etc, these all take time, and you don’t want to begin from a cold, standing start once you see signs of product-market fit. So focus on your customer and product, but don’t forget to plant the seeds of these future marketing needs.

On Friday, a reader sent me an email. They were suffering from “entrepreneur’s block”, where they kill off every idea as quickly as it arises. It is the opposite disease to those who fall in love with an idea, are afraid to talk to anyone about it and thus build something no one wants. In this case, the person never builds anything at all. The cure to both problems is actually the same: get outside your own head.

It is incredibly easy to kill an idea. Innovations tend to be obvious only in retrospect. There are a million reasons why startups fail, and so a million reasons never to start in the first place. But if you really want to be an entrepreneur, you need to balance intelligent caution with entrepreneurial can-do optimism. My problem in my youth was too much of the latter, which is why I so appreciate Blank/Ries.

You need to accept that it’s ok for an initial idea to be imperfect, because they all are. You need to have a core insight, but product-market fit comes from the *process*, not from the idea. Given media-perpetuated myths, I think this notion is as surprising to some as realizing that Steve Jobs practices the hell out of his speeches, rather than just being awesome due to innate, unobtainable talent.

If you have a passion for an idea, start doing market and competitive research*, but also get talking to real people. Try different validation methods (aside: part of the fun of watching this weekend Lean Startup Machine experiment was seeing the creativity and diversity of customer development tactics). So what if the idea is half-baked! Until you start customer development, no matter how much you “bake”, it will always remain half-baked! At Aprizi, we started out with a thesis and 10 related ideas. Early customer development quickly narrowed that down to 2. The wizard-of-oz alpha brought that to one, and our crude beta made us evolve even that one quite a bit. We have a lot of learning and improvement to do still — I don’t think that ever ends.

There is nothing like hearing excitement in another person’s voice to add fuel to your own fire. But if you do not hear that excitement right away, that doesn’t mean you should necessarily kill the idea. Stubborness has a place, as does flexibility to evolve. As entrepreneurs, we work in a cloud of imperfect information. We constantly face the risk of false positives and false negatives. You have to navigate with customer development, gut and vision, experience, and the help of other smart people.

*Note on market and competitive research
If you are thinking about doing a startup, it is worth thinking through your personal filters for ideas. Do you need a billion dollar business or would a niche market make you happy? Does an idea have to be bootstrappable all the way, or would you contemplate raising money (this hugely impacts the kinds of business models you can take on)? Do you care more about enterprise or consumer? Where is your domain knowledge and passion? (Note: you don’t have to be the customer. Club Penguin was built by a group of dads who really cared about what they were doing.)

I also want to stress that the existence of competition is not, in itself, a reason to kill an idea. Excessive noise and fragmentation can be troubling (although that can open up different opportunities), but competition can be a very good thing. There are advantages to having a more educated, mature market. It all depends on how vulnerabilities match up to your strengths. A big, complex topic!

Last night kicked off an interesting experiment in New York. The Lean Startup Machine is a weekend-long customer development bootcamp where participants pitch their ideas, and all 50 people break into teams around the most popular ideas. Instead of a hackathon, the goal is to do as much customer and business validation as possible, and design some kind of MVP (does not have to be a working prototype), by the end of the weekend. I think it’s a fascinating experiment, and a great way to get people out of idea-centric comfort zones.

Lean Startup Machine teams brainstorming (1am Friday night)

Eric Ries kicked it off, then Brant Cooper, Josh Knowles and myself gave short introductory talks. I walked through Aprizi’s experiences in the trenches of “lean startup”. I’ll post the deck soon with some explanatory text, but wanted to start with this slide (note: I added the last bullet this morning).

12 Tips for Early Customer Development Interviews

1. Work to their schedule: be flexible to the customer’s schedule, not your own

2. Get psyched to hear things you don’t want to hear: if you don’t do this, you might find yourself selling or convincing, or even hearing what you want to hear. Remember, the goal in this early stage is learning and validation, not a sale.

3. Disarm “politeness” training: people are trained not to call your baby ugly. You need to make them feel safe to do this. My approach was to explain that I needed their honesty so I didn’t build something nobody wanted to use, which seemed to resonate with folks.

4. Start with behavior, not feedback: start by asking about the person’s current behavior and processes in your “problem space”. Only after you’ve covered this ground should you get feedback on your proposed solution. Another interesting follow-up set of questions is to ask about competitors (even if indirect), and how those competitors were discovered.

5. Ask open ended questions: do no ask a yes/no question such as “do you like Nextag?”  Instead ask “are you interested in deals? how do you discover deals? what do you like or find frustrating in this process?”

6. Listen, don’t talk: try to shut up as much as possible, and try to keep your questions short and unbiased (i.e. don’t embed the answer you want to hear into the question). Don’t sell — that is for another time!  Don’t rush to fill the “space” when the customer pauses, because they might be thinking or have more to say.

7. Encourage but don’t influence: if you stay *too* quiet, some folks might start getting uncomfortable, thinking that they are boring you or you are judging them. You can keep things rolling with little motions of encouragement, such as nods, “I see”, “interesting”, etc. But do not say things that might steer or influence the interviewee.

8. Follow your nose and drill down: anytime something tweaks your antenna, drill down with follow up questions. Don’t be afraid to ask for clarifications and the “why” behind the “what”.

9. Parrot back to confirm: for important topics, try repeating back what the person said. You can occasionally get two interesting results through this. In the first, they correct you because you’ve misinterpreted what they said. In the second, by hearing their own thoughts, they’ll actually realize that their true opinion is slightly different, and they will give you a second, more sophisticated answer.

10. Thank them: (self-explanatory)

11. Ask for introductions: at the end of every interview, see if you can get leads to another 1 to 3 people known by the interviewee.

12. Write up your notes as quickly as possible: the details behind a conversation fade fast, so if you haven’t recorded the session, write up your notes and color commentary as soon as you can. I brain-dump into a shared Google Doc so the rest of the team can see it.  (Note: I typically have not recorded sessions, for fear of making interviewees more self-conscious or careful, but other entrepreneurs have said to me that, while it takes some rapport-building at the start, pretty soon people forget about a recorder.)

Addendum
If you are interested in this topic, check out Cindy Alvarez’s posts on finding people and conducting interviews.

Well done to Trevor, Ben, Kyle and Josh for organizing this event.  As I was walking to the subway in the wee hours of the morning, I snapped this shot of the New York Stock Exchange with my iPhone:

Aprizi has been in open beta for six weeks now.  These last six weeks have been an intense blur, a fire hose of information, and going open beta was the best possible thing we could have done.  We are making some fairly big near-term changes because of this process.  Here are 8 thoughts on the latest phase of our customer development process (see parts 3, 2, and 1).

For those new to the blog, Aprizi is building a “Pandora for shopping”.  We are creating an entertaining, personalized experience that helps people discover cool new products and stores, especially those located slightly deeper into the “long tail” of ecommerce.  While many things are currently being changed, you can play with the first version of our “minimum” beta at aprizi.com.

1. Interact with Customers However You Can
At this point, 400 people have used the beta and that has been a solid sample to accomplish our “learning” goals.  We have talked with beta users over email and phone, and have physically watched over their shoulder as they have used the site.  We wanted to answer some core questions: Do people care about what we are creating? What aspects of our MVP (and vision) are getting the strongest reaction? What will get us to product-market fit?

For the first several weeks, I tried to email every single person who signed up for the site.  I asked them to get on the phone with me, or at least answer some questions over email.  When it comes to learning from a user, in-person is best, then phone, then email, but any interaction is better than none.  As Steve Krug says, nothing beats in-person observation for pointing out UX flaws.  The number of people willing to talk to you is an important signal.

Talking to customers is addictive, and I found it to be a great stress reliever.  An excited, happy user is the best cure for stress about your startup, but frankly any customer feedback at all feels good. Substance over theory, baby. Do not be afraid to get on the phone with customers — you might be surprised at how pleased people are to talk to a company that actually gives a damn about its customers!

2. Stay Flexible
You might find that you need to give yourself a daily quota, i.e. “I will email X new customers every morning”, but keep on evaluating your time involvement. At first, I reached out to every single new user. Then time demands forced me to scale back to just the new female sign-ups, since that is our primary demographic. This week I am pretty much putting customer development on hold. The reason: we have made our decisions on what comes next for the business, and now we need to roll up our sleeves and get there as quickly as possible.

3. Don’t Optimize for the Wrong Things (or “let some stuff suck”)
During a beta process like this, you are painfully aware of the limitations of your crude product.  While we worked hard to fix immediate technical and UX issues, we tried not to lose ourselves in optimization.  Making the MVP really solid was not our goal; finding product-market fit was our goal.  If a problem was getting in the way of our learning, we fixed it, but many things were left on the backlog.  This is easier said than done, because as you talk to more users, you start hearing the same problems over and over again.  Between your pride and a natural instinct to make customers happy, you will feel a massive urge to FIX it.  One of the great things about having a co-founder is you can keep each other’s well-intentioned, but poorly-prioritized impulses in check.

4. Ruthless Prioritization
As a small startup, your eyes will always be bigger than your stomach.  This is actually a good thing, because it forces you to be absolutely ruthless in focus and prioritization.  At Aprizi, we have a massive list of ideas and improvements.  We have made a conscious decision to back-burner development and marketing effort tied to user growth, while we focus on product-market fit.

This can require some forbearance in your discussions with outsiders.  I cannot tell you how many people have said “you should add [insert 'share links', FB Connect, Twitter Auth, etc]“. All in good time. We have to get the core product right first. Anything else is putting the cart before the horse.

“Prioritization for Pirates”: One interesting exercise is to categorize your backlog of development work using Dave McClure’s AARRR “metrics for pirates”.  For each piece of work, label it one of the following: Acquisition, Activation, Retention, Referral or Revenue.  Think through which of those AARRR issues are most critical to your business at your current stage.  Make sure you include non-customer-facing work like infrastructure needs.  I’ll note that analytics / instrumentation dev work does not fit neatly into this exercise, but it is obviously something that cannot be ignored.  Arguably this exercise was overkill for our stage, but it still helped me think through our business in an interesting way.

5. Independence & Judgement
Your job is not to abdicate vision in exchange for market feedback, but rather to synthesize that feedback with your vision.  You internalize, synthesize, hypothesize and test, all in the name of charting a better course for your business. In some cases, you might find yourself disagreeing with users.  Be careful, but don’t be afraid to stick by your guns.

For example, I’ve had people saying extremely vehemently, “if you are going to show me a recommendation, you *must* show me price,” or “your default really should be a traditional grid view, not this carousel.” In both cases, I disagree. I have a vision of a shopping entertainment experience after watching and talking to a whole bunch of users. I believe that I’m right.

You might notice that I have made no mention of split testing.  We do not have the time or resources to do split testing.  Some might consider this heresy, but I believe split testing will be really important for the *next* phase of this company, not this current phase. We are still more in the realm of qualitative feedback and judgment calls, not quantitative optimization.

6. Exercise Patience
I think patience is a big part of the lean startup methodology. While time feels like the enemy when you are bootstrapping, don’t leap ahead with your first, immediate conclusions.  It takes time to internalize all the feedback you are getting from the market, think through possible paths, chase a number of new hypotheses, and lock down firm decisions for how the business needs to evolve.

While our long-term vision of a “personal shopper” has remained remarkably consistent since Aprizi’s founding, this open beta has brought major shifts to our near-term priorities, thinking, and even our core business model. In weeks 1 and 2, we had guesses. By weeks 4 and 5, we were feeling confident on our directional and design changes.

7. Continually Re-Evaluate (except when you shouldn’t)
At the start of most weeks, I try to take a blank slate approach, and think, “if I were starting today from scratch, what would I do?” At the end of every week, I try to write up a summary of the week to our two F&F investors.  Without getting too deep into the weeds, I try to cover the major accomplishments and learnings of the week, immediate challenges, and the questions we are trying to answer for the business. I find it to be a great mental exercise.

Product development work always takes longer than you want, so take advantage of this reality and keep on re-evaluating your conclusions.  However, there are times when you need to put everything else on hold, hunker down, and do nothing but product execution. That is where we are for the next several weeks.

8. Talk to Anybody & Everybody
I have not just spoken to beta users during this process. I recommend talking to other entrepreneurs — they are a phenomenal sounding board for ideas. While we won’t look for funding for a few months, I continue to have the occasional meeting with investors because we learn so much from their questions. I should note, since this can be a topic of debate among entrepreneurs, that I have really enjoyed meeting with many of the younger VCs in the New York area — some great feedback. I also continue to have informal meetings with potential distribution partners, because those future needs might affect our current design decisions.  I learn something from absolutely every conversation.

Conclusion
I haven’t spent much time talking about our specific decisions (more on that another time), but I can say that I am 200% glad we opened up the beta when we did.  The last 6 weeks have improved focus and clarity.  We have made some important changes to product design and even our monetization plans.  I am more excited about Aprizi than ever, and my confidence is very much tied to the rigor of our customer development work throughout our short lifespan as a company. Onward and upward!

Aprizi is about to enter that crucial phase of open beta, so I wanted to pause and write down the latest installment in our customer development journey (part 1 and 2).  For anyone new to the blog, Aprizi is building a personalization engine for online shopping — a Pandora for e-commerce, if you will. My thesis is that the intersection of two trends, information sharing and information overload, will enable both the creation and the demand for personalization and intelligent filtering services.

Our customer development journey so far has consisted of the following steps over the last few months:

1. 1-on-1 customer conversations with 2 parts: open-ended questions about shopping behavior, and feedback on tangible product ideas (note: I tried one focus group and disliked format)
2. an early survey of about 50 consumers
3. discussions over paper mockups (really helps focus the conversation when it is time for feedback)
4. follow-up discussions around user experience on our crude alpha website

For our “alpha”, we wanted to discover differences between people’s statements and actions.  Rather than wait for a lot of technical infrastructure, I asked our alpha testers a few questions and played “personal shopper”, i.e. I manually researched 8 recommendations for each person. Testers then browsed the recommendations on a simple website.  Our goal was to test out:

1. did people give a damn about this kind of service?
2. what kinds of things did they click on and get excited about?

We got great feedback and learned some things that did not come up during paper testing that changed our thinking and approach. For example, we had separated deals vs product suggestions, and had even contemplated a focus solely on deals. Post-alpha, not only did we decide to do both, but we decided not to treat them differently at all.

This alpha process was, of course, entirely un-scalable, but one of the best things we did was start testing with manual labor behind the scenes rather than waiting for automation (Max Ventilla, a co-founder of Aardvark, recently wrote a great post about that as well).  After we learned some core take-aways, we ceased alpha testing. My opinion was that we learned what we could even with just a small number of people, and the next real wave of learning would come with beta (which still has manual processes, but will be considerably more scalable).

It is worth noting that I have limited my customer-facing time and focused almost entirely on product for the last few weeks (I do everything but code). If our focus is on learning and validation, and the next big dose of that needs to have the beta in place, then everything else gets second shrift. On that note, every few days Liz and I have been reviewing our coding task list with a scalpel to make sure we’re really stripping out everything but the true necessities.  We *have* put time into cleaning up the UI, but we felt that we needed a decent baseline of quality given that our target audience is not techie.

Open or Closed Beta?

Our forthcoming beta is going to be completely open.  I had originally been thinking about doing a gmail-like beta program, where people could invite others, and then I changed my mind.  None of my initial reasons held up after a sharp poke with a stick.  Here is my thinking:

1. we’re not celebrity enough that “exclusivity” will be a huge factor
2. we’re not celebrity enough (nor pursuing any PR at this point) that we really need to worry about scalability breakage
3. the first version of the app will be crude and bad, but so few people are going to see it, it doesn’t matter from a brand perspective
4. far more important is validation and learning, which means maximizing conversion rates, which means letting anyone in who we are blessed enough to *get* to our home page!

Does Lean Startup Work?
What have we gained over the last 4 months in getting to this point?  It is too soon to say what our “product market fit” will look like, but I can say than thinking about lean startup ideas has sped us up.  It helped us focus on the most important things, which is *beyond* critical when you are a tiny, bootstrapping team.  When your brain is swirling night and day around customers, market, distribution, partners, monetization, competition, investment, etc. it is great to have the lean-startup framework to come back to and cut through the mental noise.

(btw, I think the “lean startup” framework should be judged on how it enhances speed, prioritization and focus, not the ultimate success or failure of a particular company)

Have we done everything perfectly? No. We could have done some things faster, and we lost time early in the process  validating some technical hurdles rather than ruthlessly focusing our coding effort on customer value proposition. I also think I should have prioritized landing page tests earlier. But we’re here, we’ve learned a lot, and we’re about to learn a ton more, good or bad.

If you want to get an email when open beta kicks off (and I hope you do!), please sign up over at Aprizi or just write me directly (giff.constable -at- gmail).

Update: Customer Development Journey, Part 4

Here are a few thoughts on product-market fit that came up when I was chatting this evening with Brant Cooper of Market by Numbers.

Revenue & Testing
To me, “lean startup” and “product-market fit” boil down to rigorously and continuously testing your assumptions as early as possible, and holding off heavy investments in scalability and growth until you feel confident in the drivers of your business.

Your revenue model is one of those assumptions, but it doesn’t have to be the very first thing empirically proven.  I am thinking specifically of consumer-focused services with variations of freemium (for example, Evernote or Zynga) or business models which separate the end-user and the revenue source / payer (for example, Daily Candy or Google).

In these cases you can build early confidence in your plan with customer development conversations (in the latter case above customer = both users and payers), as well as close observation of customer (or payer) behavior with similar products.

Take the example of a game which makes money from virtual goods.  You have a simple foundation — the game must be fun. Without that, you have nothing.  So you can first test fun, then virtual goods compulsion loops and most likely viral behavior, and then finally whether people will indeed pay for those virtual goods.  The key is that you are testing assumptions all the while, rather than waiting to do it all together after a longer product dev cycle. You are carefully choosing the order of what you test (for example, a more utilitarian product might want to test monetization earlier).  You are keeping an open mind and an eye out for the necessity to change/pivot.

The RIGHT Product-Market Fit
I like Sean Ellis’ target that 40% of your users should say that they would be “very disappointed” if they could not use your product.  It feels like a good stake in the ground.

I think that you must consider customer demographics (UPDATE: I’m using the word demographic loosely here — it’s about understanding customer commonalities and segmentation) when you do this kind of testing. Who is making up this data set and how does that fit into your assumptions?

If you were shooting for teens or 30/40-something women and you end up with the TechCrunch set all jazzed up, you need to ask yourself: “Should I give up on previous assumptions and instead aim at the TechCrunch set? Do I believe that I can effectively monetize that type of customer? Will I get stuck here, or will I make a leap to other types of users?” And if those answers are not coming back positive, “Am I willing to risk losing that momentum by changing my product to make it more compelling to mainstream users today?“  As with all this stuff, there is no universal right answer. Every startup has to navigate their own way.

Chris Dixon had a great post on this called “Techies and normals“.  I don’t believe that “techie” defines “early adopter” to the exclusion of all other types of people.  Technology is too pervasive in our culture now to be that simplistic.  You will have an adopter curve among “normal” (i.e. more mainstream) users too.

Steve Blank writes about not getting ahead of yourself by designing a product for the fat part of the adoption curve, and his argument makes sense.  You do need to think about your early adopters, but within a target demographic.  As always, this is about testing assumptions and being ready to pivot, either on design or on target market, if your assumptions were wrong.

(image from tony newall on flickr)

Conversations with prospective customers is an unending process. Here is a quick update (first post) on some of those efforts with Aprizi:

First, conversations have gotten a lot more concrete as we’ve honed our thoughts. We started out talking about people’s challenges and a wide array of features. The first wave of discussions helped us narrow down and prioritize what we wanted to build. Now we are focused on reinforcing and adjusting to the following:

• Do we fully understand the customer’s online behavior and challenges/needs?
• Do we understand customer differences based on age, gender, location? (help define sweet spot)
• Do we understand how this customer is using or reacting to indirect competitors?
• How do we reach this customer?
• Are we building the right thing?

Paper mockups have been a very helpful addition. Just showing a couple of printouts of the potential website really accelerates the conversation. People can see the primary feature right there on the page and riff on what they like, dislike, or think we’re missing.

At the start of these talks, I continue to stress the desire for blunt feedback, even if negative. I think people genuinely respond to the statement that honest feedback helps us avoid building the wrong thing.

At the end of these talks, I always ask whether the product is: “I must have this now!”, “sounds interesting, might try”, or “not really for me”. (and try to read their tone and body language as they answer)

I also try to end each conversation asking if the person can introduce me to one or two more people who like to shop online and might be able to give me more feedback.  This often requires follow-up, but I’m trying to always send a thank you anyway.

I know that the reality here is that what people say isn’t always what they do, but this kind of learning is the best thing we can do while we move towards minimal viable product.

Experiment: A Group Discussion
On Monday, I tried my first group meeting as an experiment and found the dynamic a bit challenging. It was useful, but in the end, I felt like we would have gotten better info by meeting each person individually. It was no fault of the participants — they tried very hard to be generous and honest — but rather it was the format I found difficult.

To paint the scene, I met with 3 women, and my technical partner sat in to observe. I was consciously trying to avoid “group think” in the flow of the conversation, but found it impossible to pursue a line of questioning very far or dig deep to understand the true motivations of a participant. The conversation felt a lot more disjointed. It was much harder to read each individual. Because I was trying to manage the group, it was also much easier to forget things.

In one case, one women stated a worry and the others followed suit. It was on a topic that previous interviewees have not been concerned with. Now, all three could have been legitimately concerned and we’re going to assume that is the case, but we also wonder how much the second two were influenced by the first person to speak up.

So net-net, I’m going to stick to 1-on-1 interviews. Not only are they easier to organize *and* friendlier to the interviewee because you work to their schedule, but I think the results are a lot more meaningful.

A reader named Eric asked a question about customer validation / development on my “Bull doesn’t build” post, and I thought I would answer with a post.  Eric’s question was:

“You talk about your ‘focus right now is on building a solid foundation of product-market fit.’
At what point do you spend more time focusing on product-market fit then on actually developing the product? I’m finding myself torn as to when do i decide to commit? how developed should my idea, or product, be before i go out and get actual market feedback about it…”

Short answer:

1. Start learning from customers right away (and research competitors). Unless you’ve been working in a space for years or your idea is incredibly simple to prototype, assume that you don’t know enough to start designing without first learning.
2. Write down all of your hypotheses and assumptions about the problem, your solution, and your hoped-for economics (i.e. how you make money), and try to discover the truth behind those assumptions. Review this list regularly because it will change.
3. When you talk to people, start with open-ended questions and then get reactions to specific ideas.  Ask about problems and/or behavior before suggesting your ideas for solutions. Listen, don’t talk.
4. Talk to people.  You can do a survey and try techniques like “SEM to a mock up page” to test interest,  but nothing beats looking someone in the eye and reading their body language. (Link: great Steve Blank post)
5. In our case (a new consumer Internet application), I connected with ~100 people first while my technical co-founder did some technology investigations. Once we felt confident that we were on to something, we committed.  We are building our first prototype now and in the meantime I am designing UI mockups to do 1-on-1 and multi-person paper tests, and continuing to learn, learn, learn.

I’ll add that with some things, like game design, you really can just jump right into crude paper testing and go from there.  A long answer is after the fold…

Long answer:

My new company, Aprizi, is now only about 5-6 weeks old, and I’ll walk you through a bit of our process so far.

After the light bulb moment, I started by calling on two types of folks: 1. potential customers, and 2. some smart entrepreneurs and investors I knew.  The goal with the first group was to figure out whether people would actually want my service.  The goal with the second was to see gut-reactions to the idea, reality-check the technology concepts, network to folks with domain knowledge who could advise me, and discover previous attempts or current competitors which I had not discovered.

Potential customers – Since my concept is a consumer play, I needed to cast a wide net.  I either met with or spoke on the phone with people of both genders, a mix of ages, living in city / suburbia / country, and who knew me by various degrees (and asked for introductions to folks who didn’t know me).  I started with people I knew would give me honest, blunt answers, and always started a conversation stating that I needed honesty, not kindness.

I also ran a survey with Google forms, which was interesting but at this ephemeral stage I find surveys to be most useful for gathering facts rather than subjective/opinion declarations.  In many cases, I found subjective survey responses to conflict with what I then heard in an in-person interview, and I believe the latter.

Remember that your goal is maximum honesty, not to have someone tell you what you hope to hear.  Press people about whether your idea sounds like a must-have or a nice-to-have.  Frankly, if no one is saying the latter (or worse), then you are probably doing something wrong in your approach (nothing is loved by *everyone*).  Be prepared to be wrong about what the market wants, and iterate to something better.  However, there are no one-size-fits-all rules about startups — if you are doing something truly disruptive, you might have to buckle down, believe in yourself, and bull your way through a lot of initial “you are crazy”.

Investors — Knowing that we will eventually need to raise some seed capital around this, I also have been talking to a few VC and angel investors.  One of the many differences between today and the first time I started a company (am I starting to feel old?) is that I have some great folks who are willing to let me call them with an undefined idea.  VCs have a different thought process, and you can get a feel for key trigger points that cause excitement or red flags.

If you do this, be clear about how early you are, and steer your conversation away from feeling like a pitch.  If you don’t have relationships where you can bring a half-baked idea, there are early-stage firms out there that love to stay in tune with new ideas and startup activity, and might be willing to talk to you (in New York, Charlie O’Donnell with FRC, is very open to this).  As a rule with investors, you will be better off if you can get introduced by a mutual acquaintance.

Experts – While I continually examine assumptions and try to spot gaps in my knowledge, I have been networking to knowledgeable people who can provide insights.  A lot of people find startups interesting, like rooting for the new guy on the block, and are quite open to talking.

Two side notes:
1. I find it very useful to create a financial model even at this early stage because it will reveal your assumptions, knowledge gaps, and can point out flaws and pressure points in your ideas on how to make money;

2. I don’t believe that you are helped by keeping your idea secret, although there are instances where I will hold off approaching someone too early if competitive risk feels very strong.What’s Next?

I am lucky to have an amazing technical partner, and since no one can effectively code without uninterrupted time, I’m trying to let her stay focused on the prototype while I continue to learn from the market and just give her updates (and wear every hat *other* than coder).  Our prototype functionality is so stripped down that, at this point, the first version won’t really change much based on what I’m doing.

Customers — my immediate goal while we work on prototype v.1 is finishing UI mockups of different features we need to prioritize and using those for more 1-on-1 conversations as well as focus groups.  My goal with the focus groups is to primarily include people who do not know me (although I will be networking to them via people who do).  Once we have a prototype up, we will start a closed alpha/beta process and I might also take advantage of this cool user testing service, usertesting.com (which I just discovered thanks to Chris Carella).

Investors — at this point, I’ve gathered my initial data points and reactions, so I’m pausing this effort save for a few folks where I just want to get Aprizi on their radar.

Experts & Partners — I continue to network to people that can increase my knowledge and answer key questions.  I’m also having very preliminary conversations with potential partners who could be useful for distribution or other synergies.  At this point, my primary goal is not a deal but rather to understand what those partners care about — i.e. what can I bring to the table that interests them, and how does this affect my design decisions?

Startup Blogosphere — I’m also talking to other entrepreneurs and listening to the “lean startup” crowd on the Web to pick up ideas and methods that I either haven’t thought about, haven’t prioritized appropriately, or have simply forgotten.  There are lots of great ideas out there.

I hope this post has been useful — I’m sure I’ve left stuff out but it’s back to the hustle for me.  What do you think I’ve forgotten?