RAA: A Practical Open Card Sorting Study

RAA stands for: Research Article Analysis

Paper discussed:

Lewis, K. M., & Hepburn, P. (2010). Open card sorting and factor analysis: a usability case study. Electronic Library, The, 28(3), 401-416. doi:10.1108/02640471011051981

1. Purpose of the research:

This paper describes the process, analysis, results, and implications of a card sorting usability study conducted for redesigning the library Website of University of Illinois at Chicago.

2. Methods:

Totally 18 participants were recruited and 15 were completed the task at the end. Among these 15 participants, 7 are undergraduate students, 7 are graduate students, and 1 is faculty member.

The open card sorting was done individually with each participant.

  • The researchers created 93 numbered index cards. Label on each card was one of the existing or potential content from the library website.
  • The participants were allowed to create anything missing or duplicate cards where they felt the cards belonged to multiple categories, using blank index cards.
  • The participants were also allowed to discard cards that they felt redundant or named a category with “other”  or “general” for cards seems necessary but not fit into any other categories.
  • Speak aloud protocol was carried out to gain the participants’ rational for sorting decisions.
  • The researchers didn’t answer any question to define the label terminology or indicate which category should a card go.

Post-analysis using “factor analysis” was done to indicate the association of a card with a category.
3. Main Findings:

Using the methods above, the researchers got the final result of sorting these 93 cards into 11 categories, with 27 cards not sorted into any categories. Besides the final sorting result, the researchers also concluded some qualitative findings as follows:

  • Participants tend to group together cards that have the same words on them.
  • Participants were sorting not only by format but by processes or tasks.
  • The 27 cards that were not associated to any categories were due to different reasons: should be piled to themselves; redundancy; vague meaning; meaninglessness.

4. Analysis:

I found this paper particularly interesting because the up coming card sorting project. This 2010 paper not only described the project process and findings, but also did a good job reviewing previous card sorting study done by several university libraries. The qualitative findings of this paper reminds me of some essentials of designing card sorting experiment, mentioned by Gergle & Wood (2002), such as “listen to other comments about the content” and “include a brief description on each card”. Overall, it is a well-written paper with considerable details and discussions, which could be used as a good reference to our project. What I am going to dig deeper is their statistical method to combine the results. This seems to make much more sense than just eyeballing the results. I will write about different methods of combining card sorting results in a future post.



Gergle & Wood (2002), Usability for the Web: Designing Websites that Work. Morgan Kaufman.


One thought on “RAA: A Practical Open Card Sorting Study

  1. Mihaela

    Yes, factor analysis seems to be appropriate for grouping cards into categories. See Tullis & Albert for their matrix method that calculates proximity scores in between cards.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s