Defining your own categories, rather than letting AI propose a starting set from your actual data, makes sense whenever you already know roughly what you're expecting to find - a specific set of internal reporting labels, a taxonomy that needs to match last quarter's for comparability, categories tied to how your team already talks about the problem. It also puts the quality of the result almost entirely in your hands, in a way that AI-proposed categories don't: the classification can only be as clean as the definitions it's working from, and a handful of specific, easy-to-miss mistakes in how those definitions are written account for most of the muddled results people get.
Table of Contents¶
- Overlapping Categories
- Definitions That Are Too Abstract
- Missing a Catch-All for Non-Substantive Responses
- Categories Written From Your Perspective, Not the Respondent's
- Too Many Categories, Too Soon
- A Worked Example
- FAQ
Overlapping Categories¶
The most common mistake is writing two or more categories that genuinely could apply to the same response, without a clear rule for which one wins. "Product Quality" and "Missing Features" sound distinct in a category list, but a response like "it's missing the reporting tools I actually needed" could reasonably belong to either, depending on how strictly each is defined - and without an explicit tiebreaker, a classification process has to make a judgment call the definitions themselves didn't actually settle. The fix isn't necessarily merging the categories - sometimes both are genuinely useful to keep separate - it's adding an explicit boundary rule to each definition: "Missing Features covers requests for capabilities that don't exist yet; Product Quality covers complaints about capabilities that exist but don't work well," with an example of each to anchor the line.
Definitions That Are Too Abstract¶
A category defined only by its name and a one-line abstract description - "Onboarding: responses about the onboarding experience" - leaves a lot of room for inconsistent judgment calls on anything that isn't a dead-obvious match. What counts as "about onboarding" - only responses using that word, or also responses describing a confusing first-week experience without naming it as onboarding specifically? Does a response praising a helpful account manager during setup count? A definition with two or three concrete example responses, ideally including a borderline one and a clear explanation of which side of the line it falls on, does more to stabilize a classification than a longer, more elaborate abstract description ever will - the same principle covered in our guide on writing a good classification goal, applied specifically to how individual category definitions get written.
Missing a Catch-All for Non-Substantive Responses¶
Real open-ended datasets always include a share of responses that don't actually say anything usable - "n/a," "none," a single punctuation mark, a response that just restates the question back. Without an explicit category built to catch these, they get forced into whichever substantive category happens to be the closest technical match, quietly diluting that category's real signal with responses that shouldn't be counted as meaningful input to it at all. Adding a plainly-named non-substantive or no-comment category, even if you expect it to be small, keeps your other categories honest - and it's worth checking its size after a first run, since an unexpectedly large non-substantive category can itself be a useful, if slightly deflating, finding about how engaged your respondents were with the open-ended question in the first place.
Categories Written From Your Perspective, Not the Respondent's¶
It's natural to define categories around the internal structure your team already uses - department names, product module names, ticket types from a support system - and that internal structure doesn't always match how respondents actually describe their own experience. A customer complaining about "the app being confusing" isn't thinking in terms of your internal information-architecture or navigation-design ticket categories; they're describing a felt experience that might map to several different internal categories depending on which specific screen they mean. Categories defined too tightly around internal jargon or structure risk a lot of ambiguous, hard-to-place responses, not because the classification process is struggling, but because the category list itself doesn't match the vocabulary and mental model the actual respondents are using. Reading a sample of real responses before finalizing a category list - rather than designing the list purely from an internal org chart or ticket taxonomy - catches this mismatch before it becomes a whole dataset's worth of awkward-fitting responses.
Too Many Categories, Too Soon¶
A first attempt at a category list is often more ambitious than the data actually supports - fifteen or twenty finely-drawn categories sound thorough, but if half of them end up with a handful of responses each, the resulting breakdown is more noise than signal, and the fine distinctions between similar categories become harder for a classification process to draw consistently than a shorter, more clearly differentiated list would be. Starting with a smaller set of clearly distinct categories, reviewing the results, and splitting a category that turns out to be a genuine grab-bag - using the second-pass approach covered in a separate guide - produces a more reliable result than trying to nail a large, finely-differentiated taxonomy on the very first attempt.
A Worked Example¶
A nonprofit defines eight categories for classifying open-ended donor survey feedback, including both "Communication" and "Transparency" as separate categories. After running the classification, a review shows the two categories overlapping heavily - most responses mentioning wanting "more updates on where my donation goes" landed inconsistently in one or the other, since both definitions were written abstractly enough to plausibly cover that exact sentiment. Reading fifteen responses from each side by side, the team rewrites the definitions with an explicit boundary: "Communication covers the frequency, tone, and channel of contact from the organization; Transparency covers specifically wanting more detail on how funds are used or what outcomes were achieved," with one example response anchoring each. Rerunning the classification with the revised definitions produces a much cleaner split, and the two categories - now genuinely distinct - reveal that "more updates on where my money goes" (Transparency) is a considerably larger and more specific ask than general communication frequency, a distinction the original overlapping definitions had been quietly blurring together.
FAQ¶
How many example responses should each category definition include?
Two or three is usually enough - one clear, unambiguous example and one borderline example with an explanation of which side of the line it falls on does more to stabilize results than a longer list of only obvious examples.
Should I always include a non-substantive or no-comment category?
It's a good default for any open-ended question, since real datasets almost always include some share of unusable responses. If it comes back empty or nearly empty after a first run, that's a fine, informative result in itself.
Is it better to start with fewer categories and expand, or more categories and merge?
Starting with fewer, clearly distinct categories and splitting a grab-bag category later (via a second pass) tends to produce cleaner results than starting broad and hoping to merge overlapping categories after the fact, since overlap is harder to detect and fix retroactively than a grab-bag is to split.
How do I catch overlapping category definitions before running the classification?
Read your category list and ask, for each pair, whether a single plausible response could honestly belong to either one. If yes, and there's no explicit rule in the definitions for which one wins, that pair needs a tiebreaker before you run anything.
For the full classification workflow, see Writing a Good Classification Goal and Introduction to Text Analytics.