Are you asking the right questions to get what you really want?

UX is core to understanding whether you have problem solution fit, but asking questions that honestly tests your problem assumptions is difficult.

Our team attended the #Collisionfromhome summit earlier this year. One of the partners, PitchBook shared a report on the state of VC investing including the question below:

Are you surprised by the results?

Us, not really, with the leading framing of this question, of course most VCs say things are they same. If they say yes, they risk losing out on that one billion-dollar company and their idea.

What would be a better question?

How about the number of meetings they booked this quarter vs. last quarter?

If we look at a proxy of this metric, the value of VC investments YoY we see a dramatically different story, a 5% decrease in the pace of investments compared to last year. Maybe the ideas this year are not as good as they were last year, I hear you say, whilst that is totally possible, it’s more likely the VC game was affected by that giant world pandemic. Just a thought.

So, if the intention of this question was to figure out if being remote affected investing, it did a poor job.

If the intention of the question was to make entrepreneurs feel happy about their chances, or highlight what VCs would like to think of themselves then they succeeded.

Asking the right questions

The point of all this is to say that the way you ask questions determines the quality of the data you get back. I would argue that the Pitchbook question doesn’t give you a high quality signal on whether or not remote work affects investing. But it might give you a high signal on the question “what is the perception amongst investors of how COVID has affected investing”.

Each of those questions may be interesting to have an answer too, especially the correlation between the two answers. But if the stakeholder is the entrepreneur, this data is useless.‍

How can we design the right questions?

Rob Fitzpatrick goes into great detail into how to ask the right questions in his phenomenal book the “Mom Test”. The essence is, to ask the question you want to ask without making it obvious that you are asking the question. That’s why the question on the number of deals is a better question than asking for VC’s perceptions. The number’s question gets a signal on the actual actions of the person, whereas the latter reflects their subjective perception, equivalent to you asking your mother if she thinks the picture you drew is pretty (of course it is, she loves you and everything you do is brilliant!)

Wait a minute, I thought this author specialized in productive hiring?

Yes, indeed that’s what this blog is about. But asking the right questions is important not only for user research, but in interviews also. Remember the other article you read here about how 46% of companies make a bad hire because they didn’t test for the 5 skills? That’s because if you ask somebody — “do you deal well with feedback?”, they are unlikely to say “No”, whereas if you put them into a situation where they receive feedback you get a much clearer signal from the second compared to the 1st.‍

I want more examples!

At LMNS, we seek to find candidates with grit. However, we felt that Grit Scale asked very leading questions that were easy to game. For example, how would you answer the question:

“New ideas and projects sometimes distract me from previous ones”.

The word distract has a negative connotation. It feels like I lack focus, moreover, which situation does this apply to? I rarely get distracted when I am reading a Medium article, but writing this post…

This is our redesign of that question:

‍

The same question is asked without mentioning distraction at all. Instead, creating a real-life scenario that is easy to imagine, and less leading of what constitutes the “right” answer. “Yes, I answer the phone immediately because it might be an emergency” sounds just as valid as “no, I finish my work first.”

‍We invite you to try both and let us know what you think. My grit score was a lot lower for my own assessment that the official one.

‍If you are about to run some user research and are interested in creating a behavioral survey, contact us here

‍

‍

‍