View the original post
Summary: Poorly designed qualitative or quantitative research may produce invalid results. Avoid encouraging certain responses or behaviors and make sure that your study conditions and participants are representative.
Any UX-research study aims to answer general questions about our design or about our users. What percentage of our user population will be able to subscribe to our newsletter? What major usability issues will people encounter on our site? Is design A more usable than design B for our target audience? But any time we set up a UX-research study, whether quantitative or qualitative , there is danger that it will not reflect the reality we want to capture because the study is poorly designed.
There are two big types of study-design errors:
We’ll talk about each of these separately. But before we do, let’s note that validity is separate from reliability . Reliability of a study simply means that you will get the same result if you repeat the study. In other words, findings are not random. There are plenty of statistical methods to calculate the degree of study reliability, and the main way to increase reliability is to test more participants. But reliability is no good without validity : a study with high reliability and low validity is one where you get a really good measurement of the wrong thing.