Communicating about Collaboration: Depth of MediationFriday, April 3rd, 2009 by Jeremy Pickens
Thus far in our series on Collaborative Information Seeking we have explored two dimensions: Intent and Synchronization. The next dimension is the Depth at which the mediation (aka support, facilitation) of the multi-user search process occurs.
We can talk about three levels of mediation: communications tools independent of the search engine (e.g., chat, e-mail, voice, etc.), UI-level mediation, and algorithmic mediation. The first level typifies most searching currently being performed on the web, whereas the other two are more commonly found in research prototypes.
At the shallowest level, users can collaborate using existing communications tools: Instant Messaging, voice chat, email, and wikis. Every query that one partner tries, or every relevant document that the other partner finds can be posted to a wiki page, sent via email, and so forth. Clearly, facilitation by the computer of searchers activities is happening; information is being communicated across the network (email, IM) and manually stored on jointly accessible content management platforms (wikis). But those tools are external to the actual search mechanism. It requires a lot of work for a user to update the wiki every time a new query is run or a new document is found. From our perspective, since the information retrieval system itself isn’t doing anything specific to help the users, and they must help themselves using external tools, this approach could be described as unmediated.
In SearchTogether, all actions performed by all search team members are automatically logged and sorted in a shared session profile. When any team member marks a piece of information as relevant, that document or link appears in a shared view. The moment any team member views a search result, but doesn’t mark it as relevant, that information is stored as well. If that same document ever comes up as a result of someone else’s search, the system can then grey that document out, so that the team member is automatically made aware that it has already been viewed and evaluated by a collaborating partner. A searcher can also dole out unevaluated results in round-robin fashion. This enables more than one partner to participate in traversing a single ranked list. Finally, search team members can annotate their search activities, so that search meta-information is available to all collaborators.
In Físchlár-DiamondTouch, users share a common tabletop touch display in real time. Queries can be jointly formulated, with each team member adding information or perspective to the query. Results can be partitioned simply by grabbing from a common results bucket; if a search partner has already grabbed a particular result, it will no longer be in the bucket and you therefore automatically are kept from duplicating that effort and slowing your joint search down.
We call these types of collaboration systems UI-mediated. Whether the UIs are located on different computers (SearchTogether) or a shared multitouch computer (Físchlár-DT), the UI’s primary function is to facilitate effortless shared awareness of activities and results. No manual updating of wiki pages or the e-mailing of results is required. These systems also offer benefit in terms of joint maximization of retrieval effort. Depending on the exact nature of the UI tool, queries can be jointly formulated and result sets can be partitioned to share workload. However, the defining characteristic of these systems is that, from the perspective of the back-end search algorithm, all jointly-collaborating users appear to be a single entity. The search engine itself is not aware that more than one person is jointly formulating a query, or dividing up the results. Hence the name UI-mediation.
Finally, at the deepest level are systems in which the search engine itself has an active role in returning different types of results based on the presence of multiple search partners. We call these search systems algorithmically mediated. Examples include Colum Foley’s dissertation work and our system, Cerchiamo, and a variety of recommender and collaborative filtering systems.
In Colum’s work, the relevance judgments assigned to documents by one user affect (via synchronized influence) the ordering of the not-yet-seen documents in the second user’s queue, and vice versa. The Cerchiamo algorithm looks at unseen, low ranked results from user#1′s query history and combines that information with user#1′s relevance judgments to present user#2 with low-ranked documents that user#1 never got a chance to see. At the same time, the system offers dynamically changing (continually-updating) query suggestions back to user#1 based in large part on the relevance judgments made by user#2.
Common to both of these systems is that, apart from carrying out one’s regular querying and relevance-marking tasks, no additional user action is required to influence one’s partner. Even though partners are collaborating explicitly, they do not have to make the additional effort of reading every document that their partner has found to alter their own information seeking behaviors toward a better outcome on the shared task. The search engine itself does it for them. The mediation is algorithmic. Interestingly, a recent study by Joho et al found that “the concurrent condition participants actually avoided viewing documents viewed by other group members and hence didn’t gain a complete understanding of the topic” compared to participants in the independent condition. (Thanks Sharoda!) This suggests that UI-only mediation may be sensitive to how the information is presented.
Similarly, user behaviors are collected by recommender systems (typically) without requiring additional work from them, and those behaviors are stored by the system to be aggregated and compared to others’ behaviors with the purpose of making recommendations.
We should note that this dimension, unlike the other dimensions, does not contain mutually exclusive options. For example, in the intent dimension, two users cannot both implicitly and explicitly collaborate with each other; it is either one or the other. However, the Depth dimension is cumulative. Recommender systems, for example, combine recommendations (algorithmic mediation) with modifications to the interface (UI mediation). In general, a system that is mediated on one level is also tends to be mediated on all levels “above” it.