On the shoulders of giants

on

I have used the phrases “publish -> filter” and “filter -> publish” in a number of recent blog posts related to scientific publishing, but had been unable to find proper attribution to them with a casual search. While reading Kathleen Fitzpatrick’s draft of Planned Obsolescence, I came across the phrase “filter-then-publish” which she attributed to Clay Shirky’s “Here Comes Everybody.” I am adding that book to the top of my reading list right now.

Planned Obsolescence explores the process of academic, peer-review publishing and argues for revising the process to be more open. In one section, she makes a good case for reviewer reputation as a necessary requirement for open reviewing (the “publish-then-filter” model proposed by Shirky):

…in a self-multiplying scholarly commons, some kind of assessment of the material being published (or having been published) remains important, but not because of scarce resources [as in traditional publishing]; instead, what remains scarce are time and attention. For this reason, peer review needs to be put not in the service of gatekeeping, or determining what should be published for any scholar to see, but of filtering, or determining what of the vast amount of material that has been published is of interest or value to a particular scholar.

She then goes on to point out that pure technological solutions to collaborative filtering will not create the kind of publications we would like to see:

Using such new technologies for purposes of deliberation, however, requires that all members of the network be equally empowered — and in fact, equally compelled — to contribute their ideas and voice their dissent, lest the network fall prey to a new mode of self-reinforcing group-think

but there is  still a remaining challenge of “placing the advancement of the community’s knowledge ahead of one’s own personal advancement.”

There is more: in addition to discussing peer review, she tackles authorship, texts, preservation, and the role of the university itself in this ecology. I am looking forward to digesting the rest of it, as it evolves. I say evolves, because Fitzpatrick’s book itself is another interesting experiment in open publishing and open review. While her book will undergo the normal publication review process of NYU Press, she has also posted a working draft of the book on a web site using CommentPress, a WordPress plugin that allows readers to comment on specific paragraphs of a manuscript. She also talks about Noah Wardrip-Fruin’s experience of publishing his book-in-progress in this manner.

Although the basic ideas of open publishing are not particularly new, this field seems to be undergoing a Renaissance of sorts due to the combined influence of social interaction enabled by the Internet and increasing volume of publications (at least in CS and in related fields) that are straining established review processes. It will be interesting in Clay Shirky touches on this in his upcoming keynote address at CSCW 2010. I hope he does!

3 Comments

  1. I didn’t understand how the one-person one-vote model would prevent “self-reinforcing group think”.

    Overall reputation of authors and groups provides some information, but I find it pretty easy to skim a paper in an area that I’m interested in and figure out whether the authors have something interesting to say.

    I don’t always agree with my peers about which papers are interesting. In fact, the level of disagreement about what’s interesting or even methodologically sound has always stunned me when I’ve been on editorial boards, program committees, tenure and promotion committees, and funding review panels.

    What I’d find most useful is better more like this that could cross up the terminology used for the same problems in different fields (e.g. machine learning and econometrics). That is, a more semantically and conceptually driven more-like-this.

  2. I think one way that the publish-filter model disrupts group-think is by allowing dissenting opinions to be visible to the public, and thus to catalyze debate. In the current filter-then-publish review system, minority opinion (pro or con) is not made public; in the proposed publish-then-filter model it would still be available to inform debate.

    And you’re right that methodological contributions often transcend the discipline. I recently read a paper on end-user programming that described a methodology I can easily apply to evaluating collaborative search.

  3. […] to it is a singular event, it may still serve as an example of the merits and perils of the publish-then-filter model. On the one hand, Deolalikar appears to have failed in a very public way, whereas if he had […]

Comments are closed.