The Future of Search: Who Will Win The Spam Wars?

The Future of Search: Who Will Win The Spam Wars?


Vivek Wadhwa


Sometimes, all it takes is a little spark to set off a major forest fire. That is what seems to have happened with my New Year’s Day post on Why We Desperately Need a New (and Better) Google. Over the last two months, there has been an avalanche of articles echoing my post (and a few before it from notable people like Jeff Atwood), including New York Magazine, Business Insider, GigaOm, TechCrunch, CNN, and The Wall Street Journal.


I had a feeling that this would get Google’s attention. And I had the same concern as when I challenged the Russian government, once, in a Bloomberg BusinessWeek article about Skolkovo (a new tech park). I feared that Google would either blacklist me or do its equivalent of putting me in a Gulag—deliver even more spam when I search websites.


But I was delighted to get an e-mail from Amit Singhal, the head of Google’s SEO team. His message was exemplary for those wanting to learn how to handle a PR crisis. Here is part of what he wrote:


I read your post on TechCrunch yesterday and was quite disappointed by the fact that Google search failed your students at their task. My team and I treat every such failure as an inspiration to improve Google. Would it at all be possible for me to get a few queries from your students for which our algorithm failed? We will debug every aspect of our system for those queries.


He went on to invite me to visit Google to show me how they run the search system and listen to any other criticism I had to offer.


I took Amit up on this and spent hours with him; with Matt Cutts, who heads Google’s webspam team; and with their lead developers. They were incredibly open and honest. They acknowledged the deficiencies of Google search, shared ideas on how they plan to fix them, and asked for feedback.


I raised concerns that “content farms” are turning the web into a massive garbage dump, that many sites are simply replicating the content of others like TechCrunch, and that Google has no incentive to stop this because it gains advertising revenue from the spammers.


The Google developers assured me that there is a very high Chinese Wall between them and the business side of the company; that they have been instructed by Google’s executives to do only what is in the interest of users—to keep improving quality of search results and the user experience. They said they understood the issues and had many solutions to the technical problems. I questioned whether the spam problems could even be solved algorithmically; whether the only solution was a curated web-search model like that of Blekko and DuckDuckGo. They convinced me that they could, and would, win the battle.


Matt said he would post a blog, which he did, on Jan 21. In it, he explained that Google had already made improvements to make it harder for “spammy on-page content to rank highly”; had radically improved its ability to detect hacked sites, which were a major source of spam in 2010; and was about to implement a change that would directly address the issue of sites copying others’ content. Most importantly, he acknowledged that something had to be done about the “content farms,” and said that Google would.


Not surprisingly, Matt’s blog led to another avalanche of media coverage. As it turns out, the biggest content farm of them all, Demand Media, was set for an IPO this week (on Jan 26). Savvy bloggers and journalists began to question whether it could sustain its profits without Google’s support. The Wall Street Journal asked Did Google Just Make Demand Media Less in Demand?, and GigaOm wondered Did Google Just Declare War on Demand Media?


Nonetheless, Demand Media had a spectacular IPO. Its investors reaped huge bounties, with the company achieving a market cap of $1.7 billion—valuing it higher than the New York Times. So the public markets rewarded junk over quality. And they called Google’s bluff.


Where does that put us? Do we have to watch the web become one big toxic waste dump—as the spammers rake in billions of dollars? Or will Google indeed save the day?


There is an event on Tuesday, Feb 1, called Farsight 2011: Beyond the Search Box, to discuss these questions. It will be live-streamed on TechCrunch (watch for a post by Jon Orlin on that day) and is being organized by BigThink, a public online forum for intellectuals (people like Gary Kasparov, Jimmy Carter, Malcolm Gladwell, Salman Rushdie, Nouriel Roubini, and Paul Krugman). BigThink has thousands of videos on its site, which 1.5 million people watch every month.


I am emceeing the BigThink event and moderating a panel with three big players: Matt Cutts from Google; Harry Shum, Microsoft Corporate Vice President who heads Bing development, and Rich Skrenta, founder and CEO of Blekko.


Here are some questions that I plan to ask the panelists. Please share your comments below and suggest additional questions. I can’t promise I’ll cover all the topics you raise, but I will bring up as many as I can.


1. How will they save the web? Is it possible for search engines to separate the wheat from the chaff—tell the difference between content produced by regular people and large-scale junk produced by the spammers?


2. How are the engines really different? Most people can’t tell the difference between Google and Bing. Where is the magic?


3. What lies ahead? What is the future of search?


I have no doubt that this will be a very lively and informative event. There are also other great presentations such as:


- Jaron Lanier, named by Time Magazine in 2010 as one of the 100 most influential people in the world, speaking about the need for a new sustainable revenue model for search.
- Esther Dyson, in an address entitled “The Future of Search is a Verb,” speaking about how we want search to help us do something—a set of many verbs.
- Demos from Blaise Agüera y Arcas, Architect of BING maps at Microsoft; and Luc Barthelet, Executive Director of Wolfram|Alpha.
- Demos from Blaise Agüera y Arcas, Architect of BING maps at Microsoft; and Luc Barthelet, Executive Director of Wolfram|Alpha. I hope you’ll tune in.


Editor’s note: Vivek Wadhwa is an entrepreneur turned academic. He is a Visiting Scholar at UC-Berkeley, Senior Research Associate at Harvard Law School , Director of Research at the Center for Entrepreneurship and Research Commercialization at Duke University, and Distinguished Visiting Scholar at The Halle Institute for Global Learning at Emory University. You can follow him on Twitter at @vwadhwa and find his research at www.wadhwa.com.