From search engine to RAG: Re-understanding information retrieval

This post documents my re-engagement with information retrieval.

In the early days, I gave up in-depth research because I felt that the technical knowledge of search engines was too huge. Later, I started to implement RAG with a large language model, and only then encountered problems such as file splitting, query rewriting, and reordering again.

These terms are not new technologies that suddenly appeared in RAG, but problems that have been accumulated for a long time in search and information retrieval, and are re-visited in new application scenarios.

This also forms a good echo with the existing Passage Retrieval article on the website: on one side is how the search results understand the paragraphs in the document, and on the other side is how the application organizes the document into knowledge that can be retrieved by the model.

The original post also mentions an early Google Search talk shared by Jeff Dean, as well as an AI-assisted summary.

What is preserved here is a learning context, and the original summary is not regarded as a complete technical specification of today’s search engines or RAG systems.