How Many Developers Do I Need?
At Evans Data we do many custom research projects for product managers wanting to shape their tools to best fill the needs of developers and for developer relations program managers who want to please their users and benchmark their programs against competitors. When putting together these research projects, we always get asked “How many developers do we need in the survey sample?”
The answer depends on how accurate you need the results to be. The larger the sample is, the lower the margin of error and thus the greater your accuracy is going to be. However, this is not a linear correlation. It takes very few additional respondents to move from a margin of error of plus or minus 10% to +-9%. In fact it only takes 23 individuals. However, to move from a margin of error of +- 4 to +-3 it takes 467. We’ve created a graphical counter that shows you how this works dynamically here: http://bit.ly/1fR9dWP
Doesn’t the size of the population we’re studying make a difference? Not really. The mathematics of probability proves the size of the population is irrelevant unless the size of the sample exceeds a few percent of the total population you are examining. In the case of virtually every type of software developer the population is large enough that the rules of probability apply.
One thing that is much more important than total respondents is the way they are recruited, from what sources and how they are screened. Respondents that click on a banner or a link they see to go do a survey are “self-selecting” and that introduces a bias – usually towards more extreme opinions. Respondents that are recruited from sources that have a slant (think MSDN or C++ review) will skew results because they have been recruited from a biased source and they won’t be representative of the rest of the developer universe. Finally screeners must be implemented to make sure the respondents are developers and not just professional survey takers and must be hard enough that only real developers who are representative can get through.
At Evans Data we take especial care to recruit developers to our panel from unbiased sources, we recruit in a two-step way to eliminate bias, and we implement basic screeners in all our surveys in addition to ones specific to each project.