top of page
Search

Neural Search Talks [7] — Evaluating Extrapolation Performance of Dense Retrieval

Updated: Mar 30, 2023

How much of the training and test sets in TREC or MS Marco overlap? Can we evaluate on different splits of the data to isolate the extrapolation performance?


In this episode of Neural Information Retrieval Talks, Andrew Yates and Sergi Castella i Sapé discuss the paper "Evaluating Extrapolation Performance of Dense Retrieval" byJingtao Zhan, Xiaohui Xie, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma.


This more fine-grained investigation sheds new light into the qualitative differences between dense retrievers and re-rankers: when focusing only on extrapolation performance, dense retrieval shows worse performance degradation compared to other methods such as re-rankers, or BM25.


📄 Paper https://arxiv.org/abs/2204.11447

❓ About MS Marco: https://microsoft.github.io/msmarco/

❓About TREC: https://trec.nist.gov/


Timestamps:

00:00 Introduction

01:08 Evaluation in Information Retrieval, why is it exciting

07:40 Extrapolation Performance in Dense Retrieval

10:30 Learning in High Dimension Always Amounts to Extrapolation

11:40 3 Research questions

16:18 Defining Train-Test label overlap: entity and query intent overlap

21:00 Train-test Overlap in existing benchmarks TREC

23:29 Resampling evaluation methods: constructing distinct train-test sets

25:37 Baselines and results: colbert, splade

29:36 Table 6: interpolation vs. extrapolation performance in TREC

33:06 Table 7: interplation vs. extrapolation in MS Marco

35:55 Table 8: Comparing different DR training approaches

40:00 Research Question 1 resolved: cross encoders are more robust than dense retrieval in extrapolation

42:00 Extrapolation and Domain Transfer: BEIR benchmark.

44:46 Figure 2: correlation between extrapolation performance and domain transfer performance

48:35 Broad strokes takeaways from this work

52:30 Is there any intuition behind the results where Dense Retrieval generalizes worse than Cross Encoders?

56:14 Will this have an impact on the IR benchmarking culture?

57:40 Outro


2 views
bottom of page