Relevance feedback as a form of deep mediation


In their recent ECIR paper, Joho et al. explored the effectiveness of several mediation techniques around relevance feedback in supporting collaborative search. They ran simulations based on TREC HARD topics queries elicited in an earlier study, and found that certain techniques were effective at increasing recall during (simulated) search sessions consisting of 20 queries.

They found that either individual or pooled relevance feedback improved simulated performance, that performance improved as theĀ  numberĀ  of team members increased from two to five (although marginal improvements dropped after the second user), and that there was no difference between pooling relevance judgments and keeping them separate for each user. They did not observe any benefits to clustering results prior to assignment to users.

These are all important questions that need to be understood both in simulations and in actual use. This research serves to inform the design of algorithmic mediation for collaborative search in the presence of symmetric roles. Other algorithms will be required to support asymmetric roles such as the prospector/miner combination implemented in Cerchiamo.