Evaluating collaborative search interfaces


Late last year Hideo Joho, David Hannah and Joemon M. Jose published a paper that described an experiment in collaborative exploratory search. They compared teams of pairs of searchers in three conditions — independent (not collaborative), collaborative without communication, and collaborative with communication. This paper is interesting for several reasons, not least of which is that it made an attempt to quantify the effects of collaboration on search performance, an important subject that has not yet received adequate attention.

They built a system that allowed searchers to share results of their activities, in a manner similar to SearchTogether. Users were able to see each others’ queries, which documents each person read, and their judgments of relevance (i.e., which documents were “bookmarked”).  In terms of our model of collaborative information seeking, their system had synchronized data, UI-level mediation, and, of course, explicitly shared intent.

Unlike evaluation of  SearchTogether, this paper applied objective measures of performance to assess the effects of collaboration. While they found an increased diversity of search terms and a reduction in redundant judgments of relevance in the collaborative conditions compared to the pooled individual results, they did not observe any differences in recall or precision among the experimental conditions.

Joho et al. speculate that one issue that degraded collaborative teams’ performance was the overhead of communication using the chat  channel built into the experimental system, and another was the strategic decision not to duplicate effort:

In the collaborative conditions, participants tended to avoid viewing those documents that have already been seen by the team member. In other words, participants did not learn about a topic from those documents bookmarked by the partner. This might cause them to develop a partial view of the topic space in the collection.

These findings suggest that UI-level mediation needs to be designed carefully to enable the the advantages of collaboration without compromising each person’s cognitive activity. Interestingly, the authors suggest a number of improvements to their system such as flagging queries as useful and supprting query reformulation. To achieve some of these capabilites, a deeper level of mediation may be required. Although this work was done in parallel with our Cerchiamo system, there are interesting similarities between the authors’ suggestions and the kinds of influence (document ordering for RSVP judgments of relevance, and query term suggestions basde on relevance feecdback) betwen searchers implemented in Cerchaimo. It will be interesting to explore the specific contributions of these deeper mediations on the overall effectiveness of collaborative search systems.