Should IR Objective Functions be Obfuscated?

on

I have a question. It’s a general question, directed at anyone and everyone.

When one is building an Information Retrieval system, one uses target objective function(s) that give an indication of the performance of the system, and designs the system (algorithms, interfaces, etc.) toward those targets.  Sometimes, those functions are open and well understood.  Other times, those functions are proprietary and hidden.

My question is: Does it do the users of an IR system a service or disservice to hide from them the function that is being optimized?  Or is it completely neutral?  In other words, does the user have to understand, or at least be given the chance to understand, what it is that the system is trying to do for them in order to get the best value out of that system?  Or can a user get results just as good without having to have a clear mental model of what the retrieval engine is trying to do?  In short, does it matter if the user does not understand what the system is trying to do for him or her?

Can someone point me to research that may have looked at this question?  If one were trying to publish original research on the topic, how would one go about designing an experiment in which both (1) this hypothesis is tested, and (2) done so in a way that generalizes, or at least hints at possible generalization?

3 Comments

  1. There are some related research on explaining results of recommender systems to users by the UMN folks that was useful for framing this problem. Users clearly liked explanations why something was recommended to them. Given that many recommender systems are not designed with relevance feedback in mind, I think it is less clear that users benefitted from the explanation by being able to formulate better queries or was able to steer the system toward better results.

  2. Ed, I also mean things like in web search, how one of the things that is optimized under the hood is recency. All else equal, the search engine will show you recent results, first.

    So if you’re doing a search for something that happened in 1999, and never find it because the sort order is wrong, does that matter? If the user doesn’t understand that he or she is seeing more recent results first, if that is not in the user’s conceptual model of what is happening? Does it matter if the user does or does not understand that this is what is happening?

  3. Does “A Case For Interaction: A Study Of Interactive Information Retrieval Behavior And Effectiveness by Koenemann and Belkin count?

    http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.47.9803

Comments are closed.