Ironically, the multiple overlapping panes that are so appealing in this design echo windows of desktop user interfaces, something that most apps have avoided to date.

The tool also includes a search interface, but I could not find a way to save searches persistently. One of the main uses of search for me is to follow conference-related tweets, and it’s nice to keep those around for longer than Twitter’s ever-shrinking search window permits. With the current limit of something like 4 or 5 days, early tweets from a major conference will not be accessible by the conference’s end. Thus it’s useful to have clients (such as TweetDeck) that can cache search results and supplement them with new tweets.

In short, it’s nice to see the iPad application user interface evolve to incorporate new styles of information presentation; I hope these kinds of interactions will make it into other applications. On the other hand, I am not giving up on other Twitter apps yet.

One of the highlights of the 2010 conference was a debate between the system camp and the user camp about the value of simulating users. (See Saturday August 21 in the program.) This was a reprise of the theme of a workshop held at this year’s SIGIR conference, this time on the other side’s turf.

Leif Azzopardi, simulator

Ian Ruthven, denier

I cannot possibly do justice to the rhetorical prowess that was displayed in this debate, and I am not going to take sides, either. In truth, both sides made good points and silly points, and the audience’s vote toward the user-centered approach more likely revealed their a priori biases rather than the cogency of arguments from either side.

On the other hand, one point that the anti-simulation side made abundantly clear was the unpredictable variability of people’s behaviors when engaged in information seeking. This variability led me to associate their perspective with that of Niels Bohr, who argued for lack of determinism in the physics of subatomic particles. That contentious (at the time) position became known as the Copenhagen Interpretation of Quantum Mechanics. While the analogy isn’t perfect, I tend to think of the IIiX crowd as representing the Copenhagen Interpretation of Information Retrieval.

I’d like to expand his second point a bit.

The effectiveness of modern search engines may have lulled people into the implicit belief that if Bing (or Google or Yahoo!) can’t find a document in the top few results, then that document doesn’t exist. This assumption may be true for really common information (such as what Britney Spears is up to at any given moment), but many kinds of information are not  readily findable using a typical search engine. Sometimes, as Daniel Russell likes to document on his blog, it’s a matter of phrasing the query appropriately; but often you have to think more broadly about the collections you search. (For more on this, see Daniel’s Russell’s HCIR 2010 Keynote slides.)

Web search technologists sometimes talk about the Deep Web, that is, information that is accessible through a web browser, but not findable through a horizontal search engine (Google, Yahoo!, Bing, etc.). There are many databases (e.g., genealogical data) that you can find through the web, but whose contents are not indexed along with all the other web documents.

There are several reasons why this information is not available through a single search interface: database owners may want to charge (or otherwise control) access to the data, the data may be of only specialized interest, and the data may have a well-defined structure that a horizontal engine will not be able to manage effectively.

Again, in the genealogical realm, one can search using all sorts of (partial) information such as names, dates, places, and relationships, but these values are logically related by very well defined rules that don’t apply to other kinds of data. The search engine has to have specific knowledge about the nature of similarity of the records being searched; simply lumping names and dates into a “document” will not improve the quality of search results.

So one lesson for searching more effectively is to understand the nature of relevant collections, and to recognize when to use a web search engine as a search engine, and when to use it as a finding aid to find other collections.

Another thing to keep in mind as you’re searching databases is how much to you trust the ranking function: modern search engines like to estimate the quality of the match to improve precision, and that works well as long as the estimates are accurate. In some cases, however, it may be useful to look deeper in the result set if there is reason to suspect that the ranking algorithm isn’t giving you what you expect. This is particularly true in the case of genealogical data where there are multiple competing dimensions for similarity and the data is noisy or even wrong.

There is a well known (at least to me) joke about a guy looking for his glasses not where he lost them but under a streetlamp. Why? Because that’s where the light is. We often make the same mistake when picking a horizontal web search engine for our searches, rather than searching a more focused collection. Knowing where to look is one the skills that define good reference librarians, a skill that can be learned and practiced by anyone.

Nick (a former FXPAL Intern) is exploring the design space of interactive, semi-automated visualizations that can be put together quickly and yet leverage the kinds of interaction design characteristic of computer games.

The idea is to enable a journalist to illustrate a story with an interactive data set that would engage readers and cause them to learn more about the data. The challenges include how to create an engaging experience without the kind of authoring process that goes into creating games, and how to translate that engagement into learning.

Nick’s visualizations (which you can try here) are based on the distributions of various kinds of data over the map of the US. The interactions are interesting, but I think more work remains to be done to make them more engaging and more informative.

My take on improving engagement is to add a competitive aspect. Rather than deriving the reward structure of the game from the data, it might be better to make it competitive: you earn points in solving tasks, and system ranks you compared with some other recent players. Of course this requires a more sophisticated back end that can keep track of scores, but it’s not too hard to build these now.

To make the interaction more informative, the tasks should be designed so that they cause people to form and then test hypotheses. Nick’s data includes such health-related variables as adult smoking, binge drinking, diabetes and obesity rates (among others). One possible class of interactions that might be both challenging and give insight into how unevenly these kinds of variables tend to be distributed is to find regional outliers: states or counties for which the reported rates are significantly different from their neighbors. The task could be made more interesting by starting with a coarse sample of the data (state level, for example), and then drilling down into more specific regions, and then counties. My guess is that state-level averages mask a lot of county-level variability, which might make for some interesting sleuthing.

A second class of hypotheses meaningful for these kinds of data revolves around correlations: Where are two variables (e.g., soda consumed and diabetes rates) correlated (positively? negatively?), and where are they not? Is there another factor that can predict the correlation? These are basic data analysis questions that might be useful to teach to students and to newspaper readers alike.

Another way to increase engagement is to localize the data to the region from which the person is accessing the data (or to a region the user selects) to explore more regional perspectives. Starting with a person’s home town or county might make the data more meaningful and might provoke more informed hypothesis-testing.

It will be interesting to see what results Nick obtains from his explorations, and how to translate these lessons into tools that journalists and instructors can use without significant training.

This is a specific instance of a more general pattern of data use on the iPad: with each app holding on to its own data, it’s difficult to see how to manage notes and annotations across several applications that are required for one’s studies or work.

While app switching per se is not a difficult operation, it can impose a certain amount of cognitive overhead, and prevents the data trapped in various apps from being integrated. Copy-and-paste is a possible work-around, this will take the notes out of the context in which they were created, thereby diminishing their value. The reasoning behind siloed apps is sound from a platform stability standpoint, but from the HCI perspective, this separation can cause usability problems. When people’s tasks don’t map onto a single app but require multiple tools, the lack of data flow among the apps can create roadblocks for learning and productivity.

A more open platform such as Windows 7 or Android might avoid the segregation problem, but the lack of a coherent interface design standards will likely result in a system with poor overall usability as well.

This is an interesting tension between functionality and usability, between micro- and macro-level functionality, between artistic control of the platform designer and artistic freedom of the application designer. Unfortunately, the problems are sufficiently complex that they are unlikely to be resolved in this generation of devices and operating systems. We need more collective design experience to understand how to do it right.

Thirty-nine days ago,  a new mathematics site went beta, which initially puzzled me since the mathematics community already has the highly successful MathOverflow site. The difference appears to be that MathOverflow is specifically for research mathematics whereas the new site aims to be broader, allowing more elementary questions.

Overall, I think a proliferation of such sites is great, but it is also confusing. It isn’t always clear when a question is research level or not. There are questions tagged algebra or topology on the CS theory site that are pure mathematics questions. There’s a question tagged  graph theory that had been posted previously to MathOverflow. I am delighted to see that both cs.cr.crypto-security and quantum computing already are populated with a few questions, but similar questions in these areas received good answers on MathOverflow. It would be a shame if the proliferation of sites lead to less interaction between fields rather than more. I’ll be curious to see how the usage patterns play out over time.

From the web site:

The aims of the challenge are to encourage researchers and practitioners to build and demonstrate information access systems satisfying at least one of the following:

• Not only deliver relevant documents, but provide facilities for making meaning with those documents.
• Increase user responsibility as well as control; that is, the systems require and reward human effort.
• Offer the flexibility to adapt to user knowledge / sophistication / information need.
• Are engaging and fun to use.

Participants would be given access to the New York Times annotated corpus which consists of 1.8 million articles published in the Times between 1987 and 2007, and they would be expected do something interesting in searching or browsing this collection.

Several teams competed in the event, and their entries were judged by the workshop participants. The entries (available as part of the proceedings) were:

• Search for Journalists: New York Times Challenge Report
  Corrado Boscarino, Arjen P. de Vries, and Wouter Alink
  (Centrum Wiskunde and Informatica)
• Exploring the New York Times Corpus with NewsClub
  Christian Kohlschütter (Leibniz Universität Hannover)
• Searching Through Time in the New York Times
  Michael Matthews, Pancho Tolchinsky, Roi Blanco, Jordi Atserias, Peter Mika, and
  Hugo Zaragoza (Yahoo! Labs)
• News Sync: Three Reasons to Visualize News Better
  V.G. Vinod Vydiswaran (University of Illinois),
  Jeroen van den Eijkhof (University of Washington),
  Raman Chandrasekar (Microsoft Research), Ann Paradiso (Microsoft Research),
  and Jim St. George (Microsoft Research)
• Custom Dimensions for Text Corpus Navigation
  Vladimir Zelevinsky (Endeca Technologies)
• A Retrieval System Based on Sentiment Analysis
  Wei Zheng and Hui Fang (University of Delaware)

I liked Vladimir Zelevinsky’s system that constructed a faceted browsing interface on the fly: given a search term, it generated facets using WordNet relations, and then used terms obtained this way to populate the facets. Of course he got great performance using the Endeca back end, which was able to compute facet value counts quickly enough to populate multiple facets in a few seconds. Search results themselves were presented in a two-column newspaper-like layout which appealed to me with its clean look. Of course I am somewhat partial to newspaper layouts for presenting search results.

Microsoft’s entry performed automatic clustering of matching articles into several groups, and offered a pleasant UI for browsing the results. It was a much heavier system whose success depends on the quality of the clustering algorithm. In demos, it performed well for the most part, although some of the smaller category labels were a bit odd.

The winner of this competition was the entry from Yahoo that used some NLP techniques to identify references to time in the articles. These references were used to construct a variety of timeline visualizations intended to help journalists make sense of the data. I didn’t get to play with the system, but it looked like a solid piece of work, judging from the presentation.  It also introduced me to OpenNLP, a set of NLP tools written in Java. Apparently it’s quite useful for doing things like POS tagging and named entity extraction.

These presentations highlighted a successful attempt to put HCIR techniques into practice in an open-ended manner. The effort was also a success in terms of press coverage thanks to Daniel Tunkelang’s efforts, with the Yahoo entry receiving coverage from the Technology Review in an article titled “A Search Service that Can Peer into the Future,” which was also picked up by Techmeme.

I have been using iAnnotate to read and mark up papers, NoteTaker HD to draw and sketch, and Notes for more extended writing. Each works reasonably well for the narrow task; what’s missing is the integration. While I can add short text comments in iAnnotate, writing longer comments is awkward in that tool because the text window is so small and the centered text formatting makes it awkward to read. In addition, after reading and highlighting (I have given up scribbling for the most part), I need to write the review. While I could put it in a note or into the info area associated with the document, it feels more natural to write it in Notes. Of course the downside of this is that I cannot refer to my notes and annotations while I am writing the summary.

Another integration flaw is the difficulty of adding text notes to sketches made in NoteTaker or Penultimate. Yes, the point of those tools is to create ink, but adding notes or explanatory comments to a drawing is much easier to do via a keyboard than writing with one’s finger or the fat stylus. It’s doable, but a bit awkward to generate a lot of text and awkward to read a bunch of scribbled handwriting.

Rather than isolating these various operations — annotating, sketching, writing — and forcing each application to either re-create the feature or to do without it, it would be great to have a means of switching context to a text entry widget (for example) and then incorporating the typed text directly into the drawing or annotation app. Similarly, in iAnnotate, I should be able to create blank note sheets on which one could sketch or type, and either insert them into specific documents, to into a separate notebook.

My experience suggests that we need to think about applications on at least two levels: the task level (as we do now) and some sort of common sub-task such as text entry. The current set of controls seems to be too impoverished, and the applications are too siloed. One partial exception to this is the mail app: you can invoke it from another app (e.g, Photos) with content (a photo) that is then handled by the mail app. Once the message is sent, control returns to the Photos app. One difference, though, is that nothing is passed back from the mail app.

It will be interesting to see if the Android SDK makes it easier to integrate applications at varying levels of granularity, or if it will be possible to create and integrate more capable mini-apps. It would also be great to plug in alternate text entry widgets such as ShapeWriter.

More flexible interaction might be important if the iPad is to see any competition. On the other hand, done poorly this app mashup will make a real hash of the user experience. Seems like a great HCI/UX case study.

The use case for teaching involve using it an interactive presentation device, essentially a replacement for the laptop. One issue that the paper raises is how to compare the effectiveness of such a device with more traditional white- or black-board based instruction. While a computer may be effective for presenting prepared materials, more improvisational styles of presentation may be more effective on larger boards.

For students, the paper identifies recording, note-taking, and multi-tasking as the key aspects; of these note-taking is the only tablet-specific criterion. Here the advantage of the tablet over a laptop seems to be the ability to make free-form marks (drawing, annotating, etc.) although for many such operations a stylus is essential.

The devise summary at the end doesn’t identify a clear winner (which is not hugely surprising given that one of the devices is not currently available). The iPad is dinged for not having easy projection capabilities, and for the app-centered interaction model that may require considerable switching during the course of a presentation. For student-to-student interaction, the iPad does not offer great file-sharing capabilities (other than shoving documents through e-mail).

Since the Adam device is currently still in the realm of myth, not much can be said about it specifically. On the other hand, since it is an Android platform, some of the characteristics are predictable. It will be interesting to see if its somewhat more open application architecture and simpler approval process will result in more education-related applications being developed.

The older tablet PC fared worst among the lot, mostly due to its weight and older networking capabilities. Of these weight is the more inherent limitation; newer tablets have all of the networking capabilities of modern computers. A more significant limitation at the moment is the lack of an appropriate application suite to take advantage of its form factor.

While this paper could have done a better job comparing the existing devices, it is still a useful data point and a helpful literature review.  It’s particularly interesting to see that the Kindle and other similar devices, as their specialization for linear reading makes them inappropriate for educational applications.

Finally, it’s great to see these kinds of papers coming out of student conferences and class projects. This not only exposes students to the academic publishing and review process, but also exposes their ideas to people who would otherwise never get a chance to learn what’s happening in university programs. I enjoyed attending similar presentations at Stanford and UC Berkeley earlier this year, and encourage other programs to set up similar venues for reporting and discussing students’ work.

One group of original cast members has formed a comedy troupe called “Cinematic Titanic” (Joel, Trace, J. Elvis, Frank and Mary Jo). Basically, they do a live version of the original show (minus the robot puppets). Recently I caught a performance in San Francisco. It wasn’t surprising that the group was as funny as ever. What was surprising was the fact that all of the performers were holding iPads. They didn’t make any sort of announcement about it. They just sat down and started to to read from them. They have always used paper scripts — even during live performances — so I was surprised to see this revival of a 90’s era show using such 21st century devices.

I wanted to learn more about this so I contacted Glenn Schwartz, their PR person. He explained that the iPad solved several longstanding problems involving the creative process, performance and even travel.

During their creative process the cast will watch a bad movie and write down any jokes that come to mind. These are then sent to one cast member, Weinstein, who compiles them into a script. The script is then emailed to each cast member’s iPad. They view the script in a PDF viewer and may make changes, which are shared via email directly from the iPads. The PDF reader allows each cast member to highlight their part and to make notes. The immediate effect of this is a tremendous reduction in wasted paper. It also allows for a very rapid iterative process even though all the participants are in different locations.

Apparently Apple has done an excellent job designing the UX of their PDF reader. The interaction is so natural that the cast is able to use it as if they were reading a paper script (paging to appropriate sections and etc). A side benefit of the glowing screen is that each performer is self illuminated, requiring much less stage lighting, if any at all. I was surprised to learn that the iPads are not synchronized to each other or to the film. The performers simply “turn the pages” of their scripts as necessary.

The troupe’s five performers each need an updated script. These scripts are fairly lengthy (and heavy), weighing in at about 50 lbs a set. Now that the cast uses iPads they’re no longer obligated to carry all that extra weight around. They simply bring along their iPads, something they would have probably done anyway for their own personal use.

Clearly the Cinematic Titanic troupe would benefit from a more integrated solution. Imagine if our own XLibris were an iPad app that was extended to include more collaborative features. A cloud version might enable these performers to iterate through changes in an even more natural way — retaining their local changes while automagically pushing or pulling in important global changes.

I wouldn’t be at all surprised if a technology very much like this became the standard way that scripts are distributed. Xerography made quickly revising and distributing scripts possible; some form of XLibris on an internet-enabled tablet might make it even easier and faster.

This is just one example of the way these technologies enable people to work collaboratively. Cinematic Titanic’s ad hoc script writing process isn’t very different from the way researchers might prepare a paper or the way a sales team might prepare a presentation. A robust, document-centric application that supports annotation and collaboration running on a lightweight tablet might well be that killer app we’ve all been looking for.