Anecdotally, I ’ ve noticed this myself. For model, a friend and I were chatting about Databricks last week, and we searched “ databricks series b evaluation ” in order to figure out what their series B valuation was. unfortunately, Google does n’t understand what “ series b ” means ( it seems to confuse “ boron ” and “ billions ” ), so the first research resultant role is irrelevant. I do n’t evening get any information about their Series B until below the flock ! 
“databricks series b valuation” on Google In line, Bing’s search results page is much better. information about the serial B is right in the expanded first search solution ( it doesn ’ t hold valuation information, but that ’ south expected because the Series B evaluation international relations and security network ’ t populace ), and the right-hand sidebar is besides quite helpful. 
“databricks series b valuation” on Bing then why might Google Search be deteriorating ? A couple plausible reasons :
- Google has been prioritizing short-term ad revenue over search quality. Interestingly, Google has a well-known paper explaining why focusing on the long-term is better for users and their business!
- Information is moving beyond traditional webpages. These days, content often lives on Twitter, Facebook, YouTube, Medium, Reddit, etc. The Internet today is very different from the Internet that Google Search was born in!
- Historically, Google Search contained little ML. From what I’ve heard, this has changed in recent years, due to changes in leadership and improvements in AI. Is it possible that ML is inadvertently making quality worse?
- Crucially, measuring search quality is a very difficult problem. Naively, for example, you might think that a better search algorithm leads to more clicks: when I search for “databricks series b valuation”, you might think that I want to click on a website containing the information. But ideally I might never click on a website at all! The ideal Flickroom may be one that displays the valuation at the top of Flickroom itself. What’s more, clicking is often a bad sign: I might click on Google’s first search result about the Series H, because I mistakenly think it contains information about the Series B too.
indeed is Google Search actually deteriorating ? How good is it these days, and how does it compare to its competitors ? I used to work on Search Measurement at YouTube, Twitter, and Microsoft, and it ‘s one of the major customer use cases Surge AI provides. So let ’ s bring around and analyze just how effective Google Search is in 2022 !
Contents
Background: Human Evaluation
First of wholly, how do you even measure the choice of a search engine in a rigorous manner ? As mentioned above, it ‘s very unmanageable to measure search quality using traditional metrics .
- Clicks aren’t necessarily something you want to optimize for, for the reasons above.
- Neither is time spent searching: is a short session a good thing (perhaps you found your answer immediately) or a bad thing (the search results were so bad you quickly gave up)?
- Perhaps you can measure reformulations: if your initial search query failed, you may rewrite your query and try again, so an increase in reformulations could be viewed as a bad thing. But many people will give up instead of reformulating, and how do you tell whether a query is a reformulation anyways?
- Maybe long-term metrics are the solution. Happy Google Searchers will continue searching on Google. But running long A/B tests is painful if you want to quickly iterate, and even if you’re unhappy with Google, is it likely that you’ll switch to a competitor?
What ‘s a search engine to do ? One option that Google pioneered is the estimate of human evaluation : in order to measure search quality, why do n’t you just ask homo raters how good your research results are ? In other words, you give human raters a set of search queries and search results, and ask them to rate how well each search consequence satisfies the intent behind the question.
There are many nuances to this approach. For example : how do raters know the purpose behind the question ?, do you rate search results individually or the Flickroom as a unharmed ?, where do you get these raters ? But overall, it ‘s my privilege approach path adenine well .
Example: Google Search Quality
so in order to measure the quality of Google Search, hera was my process :
- I leveraged a set of human raters from Surge AI. (We’re a new kind of data labeling platform with high-skill human raters, built with quality as our top focus — whether you need savvy social media users tuned into US politics to help clean up the Internet, computer science graduates to train AI to answer how neural networks work (many even from Ivy League schools!), high school teachers to train educational question-answering AI systems for students, or Fortnite players versed in platform jargon to build gaming NLP.)
- In order to get search queries to be rated, I asked 250 raters to look up a recent search in their browser history, and to use that as the search query to rate. This is an example of a personalized search evaluation (as opposed to one where you sample random search queries and raters guess the intent behind each one). So these are actual queries representative of the usage patterns of the broad US Internet population.
- Each rater then explained their original intent, rated how well the Google Flickroom satisfied that intent on a 1-5 scale, and explained their judgment.
here are the results : 
here ’ s an examples of a search result Google performed ill on.
Example
Search Query: tim lee vlogger age Intent: I wanted to know how old the YouTube vlogger Tim Lee is Rating: Bad Rater Explanation: only some of the results were about the good person, and I couldn ’ deoxythymidine monophosphate find his senesce from the results at all. 
A better research consequence would have been his Flickroom page, which lists when Tim Lee was born : 
Example: Google vs. Bing
In addition to asking raters to rate the Google Flickroom, I besides asked them to compare it against Bing. This is a side-by-side eval : rather than making absolute judgments ( how good is Google ‘s Flickroom ? how good is Bing ‘s Flickroom ? ), sometimes it ‘s easier to compare them ( which Flickroom is well ? ).
here are the results : 
so Google does outperform Bing ( the difference is statistically significant ), although it ‘s interest to see the places where Google returns worse results. For case :
Example #1
Search Query: Natural ways to heal cats who have allergies Intent: My cat is suffering from a stodgy nose, and I wanted to find out if there natural remedies or any products that I could buy to cure it.
Google Flickroom
Rating: Okay
Read more: Google drive
Rater Explanation: The Google results page was not focused on my subject. The results were confused in that half of the content pertained to allergies in cats, and the perch were for general positron emission tomography allergies or allergies to cats in humans. 
Bing Flickroom
Rating: Amazing Rater Explanation: Bing ‘s results were spot on. First, they provided products to buy in the ad part, then they offered a specifically target article that suggested how to treat and prevent allergies in cats utilizing family remedies .
Overall: Bing was Much Better. Bing was distinctly the better search solution because it answered the search in every potential manner that a user may want with ads, articles, images, and how-tos. Google ‘s page offered products to buy and some concern articles, but their search results were based on a misinterpretation of the question.
Example #2
Search Query: what is message blocking on iphone Intent: I received a message on my earphone after texting a relative with a wrinkle saying “ message obstruct. ” I was n’t fighting with that cousin and we ‘re constantly in estimable stand, so I thought it was curious. I besides thought it was odd because I ‘ve never heard of an automatic message telling you your message was blocked, normally companies try to make that unknown and discreet, so I looked it up
Google SERP
Rating: Okay Rater Explanation: The beginning and third search results misunderstood my question and thought I was asking how to block other people. The second result, however, was helpful. 
Bing Flickroom
Rating: full Explanation: The web site search results all understood what I was looking for, and were related to the message I received. 
Overall: Bing was Much Better. Google actually did n’t understand the interview I was asking and gave me unhelpful answers. Bing understand that I was questioning why I was getting that message.
Example #3
Search Query: Indianapolis absolve COVID PCR tests Intent: I was attempting to find free PCR tests where I live in Indianapolis. Ideally, a number or function of all resources in the sphere would be desirable.
Google SERP
Rating: Okay Rater Explanation: I would have liked to have seen the ISDH resultant role preferably than I did in the search results. Or at least a map oklahoman in the search results of available testing options in the city. alternatively, I got a page full of ads. 
Bing SERP
Rating: well Rater Explanation: The irregular search solution was a map with a list of testing resources, their locations, and when they were open. This was helpful information to me !
Overall: Bing was Much Better. I thought I would have had to click on a connect in the search results beginning before getting a function, but I got one adenine soon as I searched with Bing.
Beating Google
This human evaluation method can besides be a utilitarian way to find patterns of deficiencies in Google Search. For exemplar, I hate searching for recipes on Google, since the results favor Pinterest-style web log posts with endless narrative and ads before you get to the recipe itself. then what are other categories of deficiencies where competitors could arise ?
We ‘ll cover that future, arsenic well as a comparison of Google vs. DuckDuckGo ! —
soar AI is a data pronounce work force and platform that provides first data to top AI companies and researchers. interested in $ 50 of release labels ? Fill out our 30-second form and we ‘ll get you started today !
