Kavli Affiliate: Ke Wang
| First 5 Authors: Haoyu Kang, Yuzhou Zhu, Yukun Zhong, Ke Wang,
| Summary:
Retrieval-augmented generation (RAG) has achieved great success in
information retrieval to assist large models because it builds an external
knowledge database. However, it also has many problems: it consumes a lot of
memory because of the huge database. When faced with massive streaming data, it
is unable to update the established index database in time. To save the memory
of building the database and maintain accuracy simultaneously, we proposed a
new approach combining a streaming algorithm and k-means cluster with RAG. Our
approach applies a streaming algorithm to update the index and reduce memory
consumption. Then use the k-means algorithm to cluster documents with high
similarities together, the query time will be shortened by doing this. We
conducted comparative experiments on four methods, and the results show that
RAG with streaming algorithm and k-means cluster performs well in accuracy and
memory. For massive streaming data, we find that our method behaves better than
traditional RAG
| Search Query: ArXiv Query: search_query=au:”Ke Wang”&id_list=&start=0&max_results=3