Search quality, continued

A few weeks back Udi Manber introduced the search quality group, and the previous posts in this series talked about the ranking of documents. While the ranking of web documents forms the core of what makes search at Google work so well, your search experience consists of much more than that. In this post, I'll describe the principles that guide our development of the overall search experience and how they are applied to the key aspects of search. I will also describe how we make sure we are on the right track through rigorous experimentation. And the next post in this series will describe some of the experiments currently underway.

Let me introduce myself. I'm Ben Gomes, and I've been working on search at Google since 1999, mostly on search quality. I've had the good fortune to contribute to most aspects of the search engine, from crawling the web to ranking. More recently, I've been responsible for the engineering of the interface for search and search features.

A common reaction from friends when I say that I now work on Google's search user interface is "What do you do? It never changes." Then they look at me suspiciously and tell me not to mess with a good thing. Google is fine just the way it is -- a plain, fast, simple web page. That's great, but how hard can that be?"

To help answer that question, let me start with our main goal in web search: to get you to the web pages you want as quickly as possible. Search is not an end in itself; it is merely a conduit. This goal may seem obvious, but it makes a search engine radically different from most other sites on the web, which measure their success by how long their users stay. We measure our web search success partly by how quickly you leave (happily, we hope!). There are several principles we use in getting you to the information you need as quickly as possible:

• A small page. A small page is quick to download and generally faster for your browser to display. This results in a minimalist design aesthetic; extra fanciness in the interface slows down the page without giving you much benefit.
  
• Complex algorithms with a simple presentation. Many search features require a great deal of algorithmic complexity and a vast amount of data analysis to make them work well. The trick is to hide all that complexity behind a clean, intuitive user interface. Spelling correction, snippets, sitelinks and query refinements are examples of features that require sophisticated algorithms and are constantly improving. From the user's point of view search, almost invisibly, just works better.
• Features that work everywhere. Features must be designed such that the algorithms and presentation can be adapted to work in all languages and countries. Consider the problem of spell correction in Chinese, where user queries are often not broken up into words or Hebrew/Arabic, where text is written right to left (interestingly, this is believed to be an example of first-mover disadvantage -- when chiseling on stone, it is easier to hold the hammer in your right hand!).
  
• Data driven decisions - experiment, experiment, experiment. We try to verify that we've done the right thing by running experiments. Designs that may seem promising may end up testing poorly.

There are inherent tensions here. For instance, showing you more text (or images) for every result may enable you to better pick out the best result. But a result page that has too much information takes longer to download and longer to visually process. So every piece of information that we add to the result page has to be carefully considered to ensure that the benefit to the user outweighs the cost of dealing with that additional information. This is true of every part of the search experience, from typing in a query, to scanning results, to further exploration.

The start of your search is typing in a query. A common cause of frustration is if you don't know the correct spelling of a word! Spell correction -- which seems like a simple and obvious feature -- hides many technical challenges. No common English dictionaries would ever include the correct spelling of Britney Spears, for instance (who, probably completely unbeknownst to her, has become the poster child example for this feature). We do a huge amount of analysis of the billions of pages on the web and our query logs to determine what are "real words" on the web, and what are likely to be misspellings. The system that gives you the spell correction has to, in a fraction of a second, consider a huge number of possible words you might have meant (vastly greater than any dictionary ever manually constructed) and determine if there is a more likely query you meant to type. When we are confident that you actually meant to type something else, we take a rare liberty with our search results: we try to distract you from looking at the top result on the page. The spelling correction is in your line of sight and colored a bright must-see red. Furthermore, we now make sure that nothing else on the page is red, unless it is as important to you as spelling! (so far, nothing is). The algorithms involved in spell correction are constantly getting better. They now work in a large number of languages and are even better at detecting when you have made a spelling mistake. Getting the spelling of your query right is so important that we are considering showing you the results of the spell-corrected query in the middle of the page (just in case you missed our bright red text at the top and bottom!).

Having formulated your query correctly, the next task is to pick a page from the result list. For each result, we present the title and url, and a brief two line snippet. Pages that don't have a proper title are often ignored by users. One of the bigger recent changes has been to extract titles for pages that don't specify an HTML title -- yet a title on the page is clearly right there, staring at you. To "see" that title that the author of the page intended, we analyze the HTML of the page to determine the title that the author probably meant. This makes it far more likely that you will not ignore a page for want of a good title. Below the title comes the snippet, and a key early innovation was in what Google showed for the snippet. At the time, search engines showed you the first two lines of the web page; Google, instead, showed you parts of the page where your actual search keywords showed up (information retrieval experts call this "keywords-in-context"). Showing keywords-in-context is visually simple and virtually indistinguishable from the simpler style of snippets, but vastly more useful in helping you decide which page to visit. This simplicity belies underlying complexity: when we create a snippet we have to go through the actual text from each result to find the most relevant part (which contain your keywords) rather than just giving you the first few lines.

We have been making improvements to our snippets over time with algorithms for determining the relevance of portions of the page. The changes range from the subtle -- we highlight synonyms of your query terms in the results -- to more obvious. Here's an example screenshot where the user searched for "arod" and you can see that Alex and Rodriguez are bolded in the search result snippet, based on our analysis that you might plausibly be referring to him:

As a more obvious example, we now extract and show you the byline date from pages that have one. These byline dates are expressed in a myriad formats which we extract and present uniformly, so that you can scan them easily:

For one of the most common types of user needs, navigational queries -- where you type in the name of a web site you know -- we have introduced shortcuts (we refer to them as sitelinks). These sitelinks allow you to get to the key parts of the site and illustrate many of the same principles alluded to above; they are a simple addition to the top search result that adds a small amount of extra text to the page.

For instance, the home page of Hewlett-Packard has almost 60 links, in a two-level menu system. Our algorithms, using a combination of different signals, pick the top ones among these that we think you are most likely to want to visit.

What if you did not find what you were looking for among the top results? In that case, you probably need to try another query. We help you in this process by providing a set of query refinements at the bottom of the results page -- even if they don't give you the query that you need, they provide hints for different (likely more successful) directions in which you could refine your query. By placing the query refinements at the bottom of the page, the refinements don't distract users, but are there to help if the rest of the search results didn't serve a user's information need.

I've described several key aspects of the search experience, including where we have made many changes over time -- some subtle, some more obvious. In making these changes to the search experience, how do we know we've succeeded, that we've not messed it up? We constantly evaluate our changes by sharing them with you! We launch proposed changes to a tiny fraction of our users and evaluate whether it seems to be helping or hurting their search experience. There are many metrics we use to determine if we've succeeded or failed. The process of measuring these improvements is a science in itself, with many potential pitfalls. Our experimental methodology allows us to explore a range of possibilities and launch the ones that work the best. For every feature that we launch, we have frequently run a large number of experiments that did not see the light of day.

So let me answer the question I started with: We're actually constantly changing Google's result page and have been doing so for a long time. And no, we won't mess with a good thing. You won't let us.

In the next post in this series, I'll talk about some of the experiments we are running, and what we hope to learn from them.

Posted by Ben Gomes, Distinguished Engineer