Menu
The great search stagnation? Search needs a total rethink.
I think we can all agree that digital is on an exponential growth trajectory and that digital convergence is accelerating. Old-world physical products are drawn into the expanding black hole we call the Internet, where they can be copied, stored and distributed instantly and with almost zero cost. Many familiar physical objects have been converted to digital: archival cabinets, books, vinyl and CD records, video tapes, cameras, audio mixers, and musical instruments are just a few examples. On top of that, our social interactions, behaviors and transactions are increasingly moving online. All this results in a massive increase in data volume.
It’s not easy to intuitively grasp what this expansion of information entails. Our biological brains are severely limited when exposed to too much information, and quickly become numb. It often results in decision paralysis. Furthermore, we process the world slowly and sequentially, putting us at a disadvantage when dealing with the ever-changing flows of data generated by complex systems. We need new tools to offset these problems and empower us to better extract knowledge from data.
Modern search engines can perhaps best be described as tools for locating desired information. Arguably, they are among the most important and valuable recent inventions of humanity. They amplify us in a way that would’ve required practically infinite time, many millions of librarians and libraries the size of small cities in the pre-digital age.
Just like a telescope extends our vision by allowing us to see far-off objects like stars, search engines essentially change our perceptual capabilities by supporting us in the selection and acquisition of information. While this is helpful, it is important to be mindful of the fact that search engines are biased; they include some things and exclude others, and they typically rank results in ways that are influenced by commercial interests, popularity and other factors. Another disadvantage is that the most widely used search engines need to cater to the average user, hence the feeling that even as the pool of information expands rapidly, we’re still stuck in the same shallow end as always.
One could be forgiven for thinking that the Internet is small enough to be “understandable” given the way search engines limit and distort our perception of the available information space. When we look for health, we mostly end up with the same few well-known information sources. In some cases, this is desirable, since it gives more weight to reliable sources and lessens the feeling of information overload when researching a topic. But it does not promote novelty and discovery of previously unseen relations in data.
A more serious problem that occurs particularly in large search engines is that they are exploitable in various ways. The SEO industry, for example, skews the ranking of search results so that some items are unfairly favored (given a high rank) by manipulating the content that is fed into the search engine.
In addition, advertising as a primary revenue source has led search engine innovations astray by supplying steady revenue streams that reduce the developer’s desire to change for user benefit. Practically, this led to stagnation in search interaction models and in many cases hidden bias. When we search for anything more than a single targeted answer, we have to repeatedly modify what we search for. This is re-search. For advertisers, this model is what you want (many more views) and it takes a lot to change that. Manually validating resources and re-searching is, however, very time-consuming for users.
Ever since the invention of writing, we have had to deal with the problem of organizing records in useful ways. For most of history, we relied mostly on indices, knowledgeable librarians and various tagging systems to accomplish this. We can call such methods of retrieval and storage “content-oriented.” Their biggest advantage, aside from ease of implementation, is that they give “users” a high degree of transparency, and that they make intuitive sense to everyone.
With the advent of the computer, we were able to store data in databases, often local to a single PC. (Older people will still remember dBASE and similar technologies). Although much smaller, such databases provided a similar user experience to modern-day large-scale search engines via query functions and report generation. While not as encompassing, specialized databases allow for accurate, focused search.
There are actually many very good specialized databases and similar data sources on the Internet, but finding them is a different story. Most large search engines do not list or rank other search engines on the front page. While one may wonder why, this is hopefully not for malicious reasons but instead due to the fact that specialized search engines are not indexed by “generalist” search engines. There’s no content to add since a search engine is just a tool and not “information-rich.” Note that companies such as Amazon, eBay and Airbnb (to name a few) use static page rendering in their specialized databases to ensure that they will be indexed by larger search engines. They are exceptions, however; most specialized databases are not, although it is difficult to quantify their proportion since they are hard to find.
Fast-forward to 2022 and the feeling of results saturation is greater than 5 or 10 years ago. To what extent is this impression driven by the increasing use of Machine Learning in search engine technology?
Machine Learning is more or less a cooler rebranding of statistics; and statistics, in turn, can be defined as the art of destroying data in order to make it more digestible for a human. A heavy reliance on statistics tends to privilege “average” results, which goes against our belief that the point of a search engine is to help users find information in the long tail. Search companies might yield slightly different results as they optimize their models in their own ways, but since the search market is cornered by very few large actors, this does not meaningfully increase diversity.
Average, over-saturated results lead to value loss. Nothing is to be found except what people with similar interests already know.
AI and ML will influence search significantly over the coming years. It is already driving the change from matching to mapping (vector-based) in different shapes and forms. So, we may see new ventures in AI-powered search, but we can be pretty confident that it will drive saturation if it is provided by only a few large organizations.
The future of search will hopefully bring many innovations. We can only pray this happens soon, because we are in desperate need of tools to amplify human cognition these days.
Classic 2D “paper style” interactions (e.g., with a ranking page) will likely lose their value as we move to rendering technologies such as Virtual Reality or Augmented Reality. VR comes with vast opportunities for newer and better ways of visualizing information; this will perhaps kill the keyword and link lists of today once and for all. It is likely that the growing prevalence in VR will seriously disrupt the search market as new actors innovate in this space over the next years.
Search needs a total rethink. We can only speculate what the trends and paradigm shifts that influence the search of tomorrow will be, but some of them are more likely:
Let the next 20-30 years of Internet become other than the “great search stagnation” we are experiencing right now. If you can code, then work on this problem.
