Card Sorting Before a Navigation Redesign: How Many People to Ask
A redesign is the moment to fix menus that only make sense to the people who wrote them. Here is how card sorting and tree testing work, what a 168-person study found about sample size, and how a small business can run both without a research budget.
Most website menus are organized the way the business is organized: by department, by internal product name, or by the order services were added over the years. That structure feels obvious from the inside. Visitors arrive with a task and their own words for it, and when the two vocabularies differ they guess, click the wrong heading, or leave.
A redesign is the natural point to check the structure against the people who use it. Card sorting is a research method in which participants group a site’s topics into categories that make sense to them. Tree testing checks a proposed menu by asking people where they would look for something in a text-only version of it. This article draws on a conference study by Tom Tullis and Larry Wood, guidance from Digital.gov and Digital NSW, and published tree-testing work from the GOV.UK team to explain how both methods work and how many participants they need.
Why menus drift away from visitors
Navigation problems rarely come from carelessness. They come from familiarity. The team that writes the labels knows what “Solutions,” “Programs,” or “Resources” contain, so the labels feel clear to them. A visitor sees only the word.
What goes wrong when structure is guessed
- Labels name the business, not the task. A visitor looking for “kitchen remodeling” has to guess whether it sits under “Residential,” “Services,” or “Projects.”
- Related items end up apart. Content filed by who produced it, rather than what it answers, splits topics a visitor would expect to find together.
- A redesign repeats the old map. New visuals over the same structure carry the same findability problems into a more expensive site.
How card sorting works
Digital.gov, the U.S. government’s guidance site for digital services, describes two forms. In an open card sort, participants receive a set of content items on cards, separate them into whatever groups make sense, name those groups, and explain their choices. In a closed card sort, they place the same cards into categories the team has already defined. Open sorts reveal how people think about the content; closed sorts check whether a proposed set of categories holds up. Digital NSW, the New South Wales government’s digital service toolkit, also describes a hybrid: an open sort first to find categories, then a closed sort to test the labels.
The analysis turns many individual sorts into one picture. Each pair of cards that a participant placed together counts as a match. Adding the matches across everyone produces a similarity matrix, and a cluster analysis of that matrix suggests which items belong together and how the groups nest.
The evidence on sample size
How many participants that picture needs was the question Tullis, then at Fidelity Investments, and Wood, of Brigham Young University, set out to answer in a paper for the Usability Professionals Association conference in June 2004. The study behind it was a real redesign: an online open card sort to reorganize the intranet site of Fidelity’s usability department. It used 46 cards, and 168 employees completed it.
The researchers then drew repeated random subsamples of different sizes, from 2 to 70 participants, and measured how closely each subsample’s similarity matrix correlated with the matrix from all 168. Agreement rose steeply at small sizes and then levelled off. Their conclusion was that it “may not be cost effective” to collect more than 20 to 30 participants. They also compared the resulting tree structures: on average, 69% of the closest pairs from the full study were split apart in trees built from five participants, against 35% in trees built from 40.
Jakob Nielsen of Nielsen Norman Group, reading the same data in 2004, reported a correlation of about 0.75 at 5 participants, 0.90 at 15, and 0.95 at 30, and recommended 15 for most projects, keeping 30 for large projects with more at stake. The two recommendations differ because they weigh precision against cost differently. They agree that five participants, the figure often quoted for usability testing, is too few for a card sort.
“The menu should be organized around what visitors come to do, not around how the business is arranged inside.”
— Windware Studio Practice Note
How tree testing completes the picture
A card sort suggests a structure. It does not show whether people can find things in it. Tree testing does that by stripping away the visual design and presenting only the hierarchy of labels.
The GOV.UK team described one round of this work in 2018, when it revisited the headings it uses to group types of content on topic pages. The team asked people where they would click to find a particular type of content in an imaginary task, and ran the test as a survey on GOV.UK, which gathered thousands of responses over a few days. Combined with lab sessions, the results showed that some users read “alerts” as email alerts, and that “engagement” meant little to them as a label.
The team then ran the same tasks against a revised structure, so the second round worked as a benchmark. One finding is a useful caution against changing labels on instinct: a proposed rename to “Announcements and updates” proved as clear as “News and communications,” but people took longer to choose it, so the team kept the original label.
The team also noted the method’s limits. Tree-test data does not always show why people choose a branch, and participants cannot see example content that would normally sit under each heading.
| Question | Method | What it produces |
|---|---|---|
| How do visitors group our content? | Open card sort | Candidate categories and visitors’ own labels |
| Do our proposed categories make sense? | Closed card sort | How consistently items land in each category |
| Can people find things in the new menu? | Tree test | Task success, wrong first clicks, and time to choose |
Limits and tradeoffs
The sample-size evidence comes from one study, and Tullis and Wood say so: results will vary with how similar the participants are and with the instructions they receive. Their participants were employees sorting an intranet’s content, not customers sorting a service menu. The 15-to-30 range is a sound starting point, not a guarantee.
A card sort also depends on the cards. Digital NSW advises limiting a sort to 30 to 40 cards at most, especially an open sort, because of participant fatigue. A small business site rarely has that many top-level items, but a full inventory of service pages can exceed it, so the cards should represent the content that will drive the menu.
Finally, both methods measure grouping and labels, not the finished page. Tullis and Wood make a modest but useful point here: even trees built from their smallest samples were probably closer to the full result than a structure guessed by a designer who is not a user of the content.
The studio approach: sorting before the redesign
Phase A: Prepare the sort
- Start from the content inventory and choose up to 30 to 40 items that represent what visitors look for.
- Write each card in plain words, avoiding the labels used in the current menu.
- Recruit people who resemble real visitors: past clients, prospects, or referral partners.
- Plan for 15 to 30 minutes per participant, the range Digital.gov gives for a card sort.
Phase B: Run and analyze
- Run an open sort with about 15 participants; add more if the groupings are still unstable.
- Ask a few participants to talk through their choices, since the reasons explain the patterns.
- Group similar category names and note the words visitors used for them.
- Draft a menu from the strongest clusters, using visitors’ labels where they are clear.
Phase C: Test the menu before building it
- Write five to ten realistic tasks, such as “Where would you look to book a consultation?”
- Run a tree test on the draft menu and record first clicks and success.
- Revise weak labels, then rerun the same tasks to compare.
Practical takeaways for your next website
- Sort before you sketch the menu; structure guessed from inside a business tends to mirror the business.
- Plan for about 15 participants in an open card sort, and 20 to 30 when the stakes justify it.
- Keep the deck to 30 to 40 cards at most, so participants can finish without fatigue.
- Test the resulting menu with a tree test before design and development begin.
- Rerun the same tasks after changes, so a new label has to earn its place.
Ready to Discuss Your Website?
If you are planning a new website, a service redesign, or custom development for your business, we’re here to help shape the work.
- Explore Our Services: Web Design · Web Development · Website Redesign
- Start a Conversation: Tell us about your project via our Contact Form or email us directly at
[email protected].