At Daniel Tunkelang’s suggestion, I revisited Google Squared, having written about it when it was first released. At the time, I tried a couple of queries (not a formal evaluation), and found some useful results, and some bogus ones. This time, I re-ran the same queries as before, and compared the results with my saved queries. For the query ‘airplane accidents’, the new results were considerably worse. For the query ‘acts of terrorism’, there were no initial results, but when I put in some instances (WTC attack, Oklahoma City bombing, Khobar towers, marine barracks) I got back a similar list to the one I had constructed in June.
For the airplane accidents query, I did a quick evaluation of the old and new results. The old results included 15 hits (I may have expanded the search once), of which three were descriptions of specific accidents, six were links to pages that enumerate accidents, and the rest were largely unrelated. The new results found no relevant hits, two images with no metadata from the commons.wikimedia.org site) , five lists of accidents (and two more redundant hits on one of the lists), three irrelevant pages, and one that produced a 404 error, for a total of 14 hits. It’s also interesting to note that the most useful hits were the sites that aggregated lists of accidents, but these are not hard to find with standard precision-oriented queries, particularly since many of them are sourced to the Wikipedia. Rather than searching broadly and turning up results with high diversity, the underlying queries seem to still focus on the same set of results that a regular query might return.
The ‘acts of terrorism’ query was a bit more effective, although it did include pages on marine barracks in Washington DC, the Rwandan Genocide and the Gulf War as a relevant hits, results that did not match my standard of relevance. It also included the USS Cole bombing twice, the bombings of the US embassies in Africa twice (with the same description but with different attributions: in the second case, NATO was marked as the perpetrator), and the Khobar Towers twice (once with no supporting description). It retrieved only three relevant instances (African embassy bombings, USS Cole and Bali bombing) that I had not specified when filling out the examples. Of the 13 instances returned, 10 were on topic, of these eight were either given or redundant, and three were relevant.
So overall, these numbers are not impressive. Two aspects are particularly troubling: the duplicate detection algorithms do not work very well, and the system does not distinguish aspects that were suggested to it from similar ones identified automatically. It is also interesting to note that the new version of the system no longer suggests facets of each hit by default. This is not too surprising, given the rather random results that the earlier version of the software produced.
With all these limitations, this interface might still be useful if it were possible to collect and manage results. Unfortunately, when the system is unable to determine a value for a facet automatically, it provides no obvious means to fill in the value. Rather than allowing the searcher to enter a value manually, the system offers to search for other values. But that takes you to the standard web search interface, with no way to get back to the square you are building.
It is also awkward to find other, additional, aspects once you’ve filtered or otherwise processed the retrieved set. The suggestion box at the bottom offers a list of ten items, but only one of those (Madrid train bombings) was relevant (and not redundant) for my terrorism query, and there is still no way to give the system negative relevance feedback to suggest which results are not wanted. Adding these items did not cause the system to adjust its set of recommendations, nor did typing in ‘bombings in Pakistan’ or ‘bombings in India’ yield more than one match for each.
Finally, there is no place to takes notes, something that would be useful to make sense of incrementally found results.
I am still hopeful that some of these problems will be resolved, and that this tool will be useful for discovering information. I also hope that Google integrates a means of providing feedback (other than by blogging) on these interfaces to improve them, rather than interpreting lack of use as evidence that recall-oriented search is not useful.