The search technology has moved beyond page ranking. Today the stress is more on the intent than the content and the major search engines are continuously focusing on improving the search experience. Over the past couple of years, Prabhakar Raghavan, Head of Yahoo Research has led and helped build Yahoo Research organization to examine some of the most complex problems facing the Internet. His extensive expertise in search algorithms, understanding the fundamental structure of the Web, and the social phenomena emerging from it, continues to play an important role as Yahoo strives to make itself the starting point on the Web. Here he talks about the changing technology landscape, the evolving ecosystem and the huge opportunities in the internet space.

How has the technology landscape changed recently in the area of search technology?

Internet search is no longer about document retrieval. The focus has to shift from content to intent. We are redefining the notions of accuracy and relevance. Users are no longer satisfied with running searches and – in response – seeing a list of documents. Users want to complete tasks; our goal as a search engine is to enable them to complete their tasks, rather than read documents, assimilate information and finally decide on actions. Going forward, the information trove that historically has been thought of as a web of pages must be thought about as a web of objects -- the people, places or politics discussed on those pages. Users' intentions are satisfied by juxtaposing the right objects, not pages.

What are some of the challenges at the core of web search?

The core challenge we see is in building and maintaining the web of objects, inferring user intent, building new object-based user experiences that satisfy intent, and the process for automating this new paradigm of relevance. Our plan is also to deconstruct the web of pages (the traditional view of search engines) into a web of objects, where an object is a restaurant, a person, place, etc. Along with this, we model (as best as we can infer from the Web and other sources) the relationships between these objects, with the goal of presenting to the user the objects most relevant to his task. This redefinition of relevance demands new technology and science that surely gets at the hidden semantics of web pages. These are all interesting challenges that we have shared with academic colleagues and we look forward to their ongoing innovation to advance the state of the art in this young area of web search.

Yahoo has taken the lead in opening up the API’s for others to experiment and perhaps innovate. Will this open up the eco system?

Market resists continuous proprietorship. There has to be an alternative. That which is closed will open up through a combination of forces. We have seen it happening in hardware, software and latest in the middle ware. We can totally expect the same thing happening in the case of metadata too. Innovation is not the exclusive domain of search majors such as Yahoo. Recognizing that we believe in and sustain the intellectual commons, we have opened up our search infrastructure for others to innovate on – this is our BOSS (Build your Own Search Service) program. Any startup (and we also have academic researchers participating) can access layers deep in our search technology to deliver and market-test their own vision of search experience. As an example, a search experience similar to that of cuil was built on BOSS in a matter of hours. Through BOSS we’re removing the burden of the hard engine work and letting others try their hand at search innovation. Hakia is one search engine which is built on BOSS.

If you look at the Indian ecosystem, the start ups in the Internet space are yet to succeed. Where do you think is the gap to be at the cutting edge?

The gap is perhaps more at the research side. In India, there is a lack of research culture. There is definitely no dearth in high quality talent. The question is what they end up doing. People exit the research stream early and end up taking projects with financial incentives or even taking shifts in their career. The number of students entering the Phd stream is also very low in India. We have to encourage the research culture which is present in countries like U.S. and Israel. In these countries the students from the beginning aim at being present in the world scene with their research. I was very impressed by an institute in India which enforces that the students have to have minimum 2 publications to his credit in order to get masters. This will encourage the culture of innovation and research.

Moreover these days more and more Indian startups are receiving funding. We see a lot of VC activities in India. This will force a culture of innovation and research as VC’s demand continuous innovation of the products they fund. The Indian Government should also give incentives and grants to aid research activities. At the organisation level also, we see a lot of research activities being encouraged. At Yahoo, we stress this and when we do that we are sure that some part of this work will be commercialized at a later stage. The ROI on research activities will be visible only in the long term. In India the trend has just taken off. Hence, am sure that in the next 5 years there will be lot of research activities happening from India.

What is India’s contribution to the search technology?

The greatest contribution from India to search technology is man power. Many of the best names in search technology are of Indian origin. Other than that, in a global scenario, where much of the research work in this area is collaborative in nature, it is very difficult to distil a contribution based on a specific geography. But if you look at the number of people, the number would again indicate that most of them are of Indian and Chinese origin.