Evidence

on

Those of you who’ve followed this blog and Jeremy Pickens’ blog will recall his many comments about Google’s un-Googly behavior. Recently, Benjamin Edelman actually tested the hypothesis about Google injecting bias into organic results. His post details several kinds of queries that don’t produce organic results. Which ones? Ones that are related to Google properties such as finance, health, and travel. While it’s clear why Google pushes its own properties, it seems that this behavior is inconsistent with the image it tries to project.

He also draws an interesting parallel to the airline industry:

I am struck by similarities between the favored treatment Google gives its own services and the favored treatment airlines previously gave their own flights in customer reservation systems (CRS’s) they respectively owned. For example, when travel agents searched for flights through Apollo, a CRS then owned by United Airlines, United flights would come up first — even if other carriers offered lower prices or nonstop service. The Department of Justice intervened, culminating in the rules prohibiting any CRS owned by an airline from ordering listings “us[ing] any factors directly or indirectly relating to carrier identity” (14 CFR 255). The same principle applies here: Google ought not rank results by any metric that distinctively favors Google.

It’s interesting to see how Google’s behavior has been changed by the power it has accrued over the last few years. It’ll be interesting to see if Google gets threatened with antitrust legislation. There are arguments for and against this possibility. Microsoft’s case can be seen as a precedent of sorts, both for governments going after large software corporations, and for the tactics that such corporations can use to avoid significant repercussions.

Finally, it should be pointed out that not all such meddling in organic search results by Google is undesirable. Jason Kottke explains.

7 Comments

  1. Gene — Sorry, but that “research” is a load of cr*p.

    First of all, the author clearly doesn’t understand the difference between algorithmic (aka organic) results and vertical results (called “onebox” results at Google). The algorithmic results under the vertical result box are IDENTICAL between the original query & the modified query. The author hasn’t shown any bias in these results.

    Yes, there is a link to Google’s vertical in the onebox. But then again, they don’t actually need to include a link to their competitors’ services at all. Yahoo & MS I’m sure get some traffic from those links. In fact, the MSN Money page doesn’t even appear in the top-10 algorithmic results, but is listed in the onebox.

    The author has found that the onebox results get triggered most likely when there’s an exact match between the query & some dictionary associated with the vertical. So, adding a comma, which probably isn’t a common mis-spelling or query-rewrite, fails to match the dictionary.

    And, to make any claims with a sample of 2-4 queries is, well, laughable.

  2. Jon, I agree that Edelman’s research is specious, but there is a legitimate question regarding whether users distinguish between “onebox” and “organic” results. I suspect that the vast majority do not.

    But with AdSense having turned the Web into a cesspool of spam farms, “organic” results are mostly garbage now anyway. Good riddance. Since Google is already handpicking the best links for us, maybe they could just organize them into a kind of directory we could browse? Yet another hierarchically-organized Oracle?

  3. Jon, while Edelman may overstate things somewhat, I do think he has a point. In terms of query sampling, my read of his post was that he tried a whole bunch of Google Health terms and that they behaved the same way.

    I agree with Ryan that if users cannot distinguish between a vertical result and “organic” results, then the claim that the entire result set is organic is meaningless.

    Finally, if the stats he cites regarding MapQuest and other similar services that compete with Google properties are correct, then one can make an argument that by mixing in the verticals into the organic results Google is abusing its dominance in the search realm to, in effect, tie its properties to its search. This is similar to the case the EU brought against Microsoft with respect to the Windows Media player. (See this analysis for more details.)

  4. “he tried a whole bunch of Google Health terms and that they behaved the same way”

    Of course they do! They’re all in the term list that triggers that vertical result. But, this is completely irrelevant to the discussion of whether organic results are biased.

    The distinction between organic algorithmic results and onebox results is the key issue here. Did Google ever claim that everything on their results page is unbiased? no. They only make this claim for their organic results, and this research purports to address that claim.

    Whether a user can distinguish between organic results vs. anything else is beside the point. The typical user doesn’t understand what a browser is (http://www.youtube.com/watch?v=o4MwTvtyrUQ) so my expectations are really low in that department.

    Your argument about MapQuest is taken, but note that in all the other vertical search examples given (health, stocks, flights), the competition is one click away.

  5. Jon, I am not so sure we should dismiss people’s inability to distinguish the injection of vertical results from organic results too quickly. I bet that if ads weren’t labeled as ads, people couldn’t tell them apart either. (In fact, I bet many can’t tell even with the labels that are in place, but that’s because the labels are designed to be hard to see.)

    Ultimately, the reason organic results should be preferred over manipulated ones comes down to people’s ability to make sense of the data they are getting back. Injecting verticals in line with other results (as opposed to on the side, etc.) necessarily injects bias into the result sets. If people are not aware of this bias due to the way that the results are formatted, then they are being deceived. Google’s claim that they don’t muck with organic results because injecting data from verticals isn’t considered mucking is good legalese but I don’t think that’s how most people would interpret it.

  6. The distinction between organic algorithmic results and onebox results is the key issue here. Did Google ever claim that everything on their results page is unbiased? no. They only make this claim for their organic results, and this research purports to address that claim.

    So let’s take this to its logical end: What if Google were to have a “launch vertical/onebox” for every term and phrase in a lexicon? So no matter what a user typed, there is always a mapping between that query and some onebox.

    Let’s call it “microverticals”.

    Furthermore, suppose Google starts selling (via “partnering”) the content of those oneboxes, i.e. anyone that wants to pay enough money can be the “owner” of that onebox/vertical.

    So basically you’d have a situation where the first/top result was always bought/paid for. But because that first result was always a onebox microvertical, you feel satisfied with Google’s claims that its “organic” results are pure?

Comments are closed.