Our guide on how many survey responses you need is about statistical precision - how confident you can be in a percentage, how tightly a margin of error narrows as your sample grows. Letting AI propose a starting set of categories from your own open-ended responses is a related but genuinely different sizing question. It's less "how confident can I be in this number" and more "is there enough actual variety in this pile of text for real, recurring patterns to be visible in the first place" - a question about signal density, not statistical margin, and one that behaves differently enough to deserve its own answer.
Table of Contents¶
- Why This Is a Different Question Than Sample Size
- What Happens With Too Little Data
- Practical Thresholds
- When You Don't Have Enough Yet
- More Data Isn't Always the Answer
- FAQ
Why This Is a Different Question Than Sample Size¶
Statistical sample-size planning asks how many responses you need before a percentage is precise enough to trust - and it applies whether the underlying question is closed-ended or open-ended, since it's about the reliability of a proportion, not about the content of the responses themselves. Letting AI propose categories from your data is asking something upstream of that: whether there's enough raw material - enough distinct phrasings, enough recurring language, enough genuine variety - for real patterns to be visible and distinguishable from noise in the text itself. A survey with 500 closed-ended responses to a five-point rating scale has plenty for a statistically solid percentage. A survey with 500 open-ended responses where 480 of them are some version of "no comment" or a two-word answer has a statistically adequate sample size and nowhere near enough actual textual signal for category proposal to find anything meaningful.
What Happens With Too Little Data¶
When there's genuinely too little textual signal to work with, letting AI propose categories tends to fail in one of two recognizable ways, rather than failing silently. Either the proposed categories come back extremely broad and generic - three or four categories so vague they could describe almost any dataset, essentially a sign the process couldn't find enough specific, recurring structure to propose anything more precise - or the categories come back overly specific to just one or two individual responses, mistaking a single distinctive comment for a genuine recurring pattern because there wasn't enough volume to tell the difference between a real theme and an outlier. Neither failure mode is dramatic or obviously broken-looking on the surface, which is exactly why it's worth knowing the warning signs rather than assuming a category list is reliable just because it was returned successfully.
Practical Thresholds¶
There's no universal magic number, since the right threshold depends on response length and how much genuine variety exists in your specific audience's answers, but a few practical ranges are worth knowing. Below roughly 10-15 substantive responses, category proposal is working with too little to reliably distinguish a real recurring theme from a single respondent's distinctive phrasing - at this volume, defining your own categories manually, drawing on your own domain knowledge, is almost always the better path rather than relying on pattern discovery that doesn't have enough data to discover patterns from. Between roughly 15 and 50 responses, category proposal starts to work, but treat the result as a reasonable starting draft rather than a finished, confident answer - it's worth reading through the actual proposed categories against a sample of the real responses before trusting the split. Above roughly 50-100 substantive responses, category proposal is generally working with enough material to surface genuinely recurring patterns with reasonable confidence, and the quality of the result becomes more a function of how well-written your classification goal is (see our guide on writing a good classification goal) than of raw volume.
These ranges describe substantive responses specifically - a dataset of 200 responses where half are one-word non-answers is really working with closer to 100 substantive ones for the purpose of this question, not 200.
When You Don't Have Enough Yet¶
If your current data genuinely doesn't clear a reasonable threshold, a few options are more useful than forcing AI-suggested categorization on too little material. Defining your own categories manually, based on domain knowledge or a quick manual read of the responses you do have, works regardless of volume and doesn't depend on there being enough data for a pattern-discovery process to work reliably. Waiting for a later wave, if the survey is a recurring one and volume will naturally grow, is often the simplest fix. And combining a genuinely too-small current wave with previous waves of a similar, consistently-worded question - if one exists - can sometimes get you over the practical threshold without waiting for an entirely new round of data collection, provided the combined set is treated honestly as a blended, multi-wave sample rather than presented as a single point-in-time read.
More Data Isn't Always the Answer¶
Past the practical threshold where category proposal has enough to work with, adding substantially more data has diminishing returns for the categorization step specifically, even though more data is almost always good for the general statistical confidence of whatever percentages come out the other end. A well-written classification goal working with 200 genuinely varied responses will often produce a cleaner, more useful category list than the same goal working with 2,000 responses that are, in aggregate, saying the same handful of things with more repetition rather than more genuine variety. If you're deciding whether to wait for more data before running a first categorization pass, the more useful question is usually whether you've cleared the practical threshold for pattern discovery to work at all, not whether you've maximized total volume.
FAQ¶
Does response length matter as much as response count?
Yes, in practice - a dataset of 100 detailed, multi-sentence responses generally carries more usable signal than 100 one-or-two-word responses, even though both count as "100 responses" by a simple headcount.
Should I combine multiple survey waves to reach a higher response count for categorization?
It can work if the underlying question was worded consistently across waves, but treat the combined set transparently as a multi-wave sample rather than presenting it as a single point-in-time snapshot, since combining waves can blend genuine change over time into what looks like a single static result.
Is this threshold different for defining my own categories manually versus letting AI propose them?
Yes - defining your own categories manually works at any volume, since it draws on your own knowledge rather than pattern discovery within the data itself. The threshold discussed here applies specifically to letting AI propose a starting category set from the data.
How do I know if my proposed categories came back too generic because of low volume?
If the categories feel like they could describe almost any dataset on the same general topic, rather than anything specific to what your actual respondents said, that's a sign worth checking response volume and substantiveness before trusting the split.
For the full workflow, see Introduction to Text Analytics and Writing a Good Classification Goal.